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Abstract 

We  present  several  new  characterizations  of  correlated  equilibria  in  games  with 
continuous  utility  functions.  These  have  the  advantage  of  being  more  computationally 
and  analytically  tractable  than  the  standard  definition  in  terms  of  departure  functions. 
We  use  these  characterizations  to  construct  effective  algorithms  for  approximating  a 
single  correlated  equilibrium  or  the  entire  set  of  correlated  equilibria  of  a  game  with 
polynomial  utility  functions.  We  then  exhibit  the  rich  structure  of  the  set  of  corre¬ 
lated  equilibria  by  analyzing  the  simplest  of  polynomial  games,  the  mixed  extension 
of  matching  pennies.  We  show  that  while  the  correlated  equilibrium  set  is  convex,  the 
structure  of  its  extreme  points  can  be  quite  complicated.  In  finite  games  there  can 
be  a  superexponential  separation  between  the  number  of  extreme  Nash  and  extreme 
correlated  equilibria.  In  polynomial  games  there  can  exist  extreme  correlated  equilibria 
which  are  not  finitely  supported;  we  construct  a  large  family  of  examples  using  tech¬ 
niques  from  ergodic  theory.  These  examples  show  that  in  general  the  set  of  correlated 
equilibrium  distributions  of  a  polynomial  game  cannot  be  described  by  conditions  on 
finitely  many  joint  moments,  in  marked  contrast  to  the  set  of  Nash  equilibria  which  is 
always  expressible  in  terms  of  finitely  many  moments. 


1  Introduction 

In  finite  games  correlated  equilibria  are  simpler  than  Nash  equilibria  in  several  senses  - 
mathematically  at  least,  if  not  conceptually.  The  set  of  correlated  equilibria  is  a  convex 
polytope,  described  by  finitely  many  explicit  linear  inequalities,  while  the  set  of  (mixed) 
Nash  equilibria  can  be  essentially  any  real  algebraic  variety  (set  described  by  polynomial 
equations  on  real  variables)  [7J.  The  existence  of  correlated  equilibria  can  be  proven  by 
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(non-zero  sum) 

(zero  sum) 

Finite  games 

Semialgebraic  set  [T7j 

LP 

LP  H 

Polynomial  games 

Semialgebraic  set  [223 

sdp  m 

? 

Tabic  1:  Comparison  of  the  simplest  known  description  of  different  classes  of  equilibrium 
sets  in  finite  and  polynomial  games. 


elementary  means  (linear  programming  or  game  theoretic  duality  [12] ) ,  whereas  the  existence 
of  Nash  equilibria  seems  to  require  nonconstructive  methods  (fixed  point  theorems)  or  the 
analysis  of  complicated  algorithms  pans].  Computing  a  sample  correlated  equilibrium  or  a 
correlated  equilibrium  optimizing  some  quantity  such  as  social  welfare  can  be  done  efficiently 
mm-,  strong  evidence  in  complexity  theory  suggests  that  the  corresponding  problems  for 
Nash  equilibria  are  hard  psun]. 

There  are  several  exceptional  classes  of  games  for  which  the  above  problems  about  Nash 
equilibria  become  easy.  The  most  important  here  are  the  zero-sum  games.  Broadly  speaking, 
Nash  equilibria  of  these  games  have  complexity  similar  to  correlated  equilibria  of  general 
games.  In  particular,  the  set  of  Nash  equilibria  is  an  easily  described  convex  polytope, 
existence  can  be  proven  by  duality,  and  a  sample  equilibrium  can  be  computed  efficiently. 

The  situation  in  games  with  infinite  strategy  sets  is  not  nearly  so  clear.  For  now  we  restrict 
attention  to  the  simplest  such  class  of  games,  those  with  finitely  many  players,  strategy  sets 
equal  to  [—1, 1],  and  polynomial  utility  functions.  Little  is  known  about  correlated  equilibria 
of  these  games,  but  much  is  known  about  Nash  equilibria.  Most  importantly,  the  set  of 
mixed  Nash  equilibria  is  nonempty  and  admits  a  finite-dimensional  description  in  terms  of 
the  moments  of  the  players’  mixed  strategies  [27]. 

This  set  of  moments  can  be  described  explicitly  in  terms  of  polynomial  equations  and 
inequalities  m  The  Nash  equilibrium  conditions  are  expressible  via  first  order  statements, 
so  the  set  of  all  moments  of  Nash  equilibria  is  a  real  algebraic  variety  and  can  be  computed 
in  theory,  albeit  not  efficiently  in  general.  In  the  two-player  zero-sum  case,  the  set  of  Nash 
equilibria  can  be  described  by  a  semidehnite  program  (an  SDP  is  a  generalization  of  a  linear 
program  which  can  be  efficiently  solved,  see  the  appendix),  hence  we  can  compute  a  sample 
Nash  equilibrium  or  one  which  optimizes  some  linear  functional  in  polynomial  time  (2U-  A 
summary  of  the  results  described  so  far  is  shown  in  Table  [T} 

Contributions  The  impetus  for  this  paper  was  to  address  the  bottom  right  cell  of  Table 
[lj  the  one  with  the  question  mark.  The  table  seems  to  suggest  that  the  set  of  correlated 
equilibria  of  a  polynomial  game  should  be  describable  by  a  semidehnite  program.  We  will  see 
that  this  is  approximately  true,  but  not  exactly.  The  contribution  of  this  paper  is  threefold. 

•  First,  we  present  several  new  characterizations  of  correlated  equilibria  in  games  with 
continuous  utility  functions  (polynomiality  is  not  needed  here).  In  particular  we  show 
that  the  standard  definition  of  correlated  equilibria  in  terms  of  measurable  departure 
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functions  is  equivalent  to  other  definitions  in  which  the  utilities  are  integrated  against 
all  test  functions  in  some  class  (Theorem  2.9).  This  characterization  does  not  have 
any  obvious  game  theoretic  significance,  but  it  is  extremely  useful  analytically  and  it 
forms  the  base  for  our  other  contributions. 


Second,  we  present  several  algorithms  for  approximating  correlated  equilibria  within 
an  arbitrary  degree  of  accuracy.  We  present  one  inefficient  linear  programming  based 
method  as  a  benchmark,  followed  by  two  semidefinite  programming  based  algorithms 
which  perform  much  better  in  practice.  The  first  SDP  algorithm,  called  adaptive 
discretization,  iteratively  computes  a  sequence  of  approximate  correlated  equilibria 
supported  on  finite  sets  (Section  3.2).  We  enlarge  the  support  sets  at  each  iteration 
using  a  heuristic  which  guarantees  convergence  in  general  and  yields  fast  convergence 
in  practice.  The  second  SDP  algorithm,  called  moment  relaxation,  does  not  discretize 
the  strategy  spaces  but  instead  works  in  terms  of  joint  moments.  It  produces  a  nested 
sequence  of  outer  approximations  to  the  set  of  joint  moments  of  correlated  equilibrium 
distributions,  and  these  approximate  equilibrium  sets  are  described  by  semidefinite 
programs  (Section  3.3).  These  relaxations  depend  crucially  on  one  of  the  correlated 
equilibrium  characterizations  we  have  developed. 


•  Third,  we  use  our  new  characterizations  to  show  that  the  set  of  correlated  equilibria 
can  be  surprisingly  complex,  even  in  simple  games.  To  this  end  we  analyze  the  mixed 
extension  of  matching  pennies,  a  polynomial  game  with  bilinear  payoffs,  in  detail.  We 
show  that  the  set  of  correlated  equilibria  has  extreme  points  with  arbitrarily  large 
finite  support  and  also  with  infinite  support  (Section  |4.2[).  This  cannot  happen  if  a  set 
of  measures  can  be  written  as  those  measures  satisfying  certain  conditions  on  finitely 
many  joint  moments,  so  it  provides  a  counterexample  showing  that  in  general  the  set 
of  correlated  equilibria  of  a  polynomial  cannot  be  described  in  this  way  (Proposition 
T5|. 


Related  Literature  The  questions  we  address  and  the  techniques  we  use  are  inspired  by 
existing  literature  in  two  main  areas.  First,  our  work  is  related  to  a  number  of  papers  in  the 
game  theory  literature. 


•  Aumann  defined  correlated  equilibria  in  his  famous  paper  [Tj,  focusing  on  finite  games 
to  establish  the  basic  properties  and  important  examples.  He  obtained  existence  as  a 
consequence  of  Nash’s  theorem  on  the  existence  of  Nash  equilibria  in  finite  games  [IS]. 


Hart  and  Schmeidler  showed  that  existence  of  correlated  equilibria  in  finite  games 
could  be  proven  directly  by  a  duality  argument  [I2j.  They  then  use  a  careful  limiting 
argument  to  prove  existence  of  correlated  equilibria  in  continuous  games  with  compact 
Hausdorff  strategy  spaces  (Theorem  3  of  that  paper).  The  germs  of  ideas  in  this 
limiting  argument  are  developed  further  in  Section  [2]  of  the  present  paper  to  yield 
various  characterizations  of  correlated  equilibria.  It  is  worth  noting  that  in  [12]  the 
authors  also  consider  part  (|T|)  of  Corollary  2.13  as  a  candidate  definition  of  correlated 
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equilibria.  They  discard  it  is  as  not  obviously  capturing  the  game  theoretic  idea  of 
correlated  equilibrium,  but  we  prove  that  it  is  nonetheless  an  equivalent  definition. 


Stoltz  and  Lugosi  have  studied  learning  algorithms  which  converge  to  correlated  equi¬ 
libria  in  continuous  games  [28] .  These  procedures  do  not  seem  to  lead  to  efficient 
techniques  for  computing  correlated  equilibria,  but  are  interesting  in  their  own  right. 
The  authors  consider  replacing  the  class  of  all  measurable  departure  functions  with  a 
smaller  class,  such  as  simple  or  continuous  departure  functions,  and  study  when  this 
yields  an  equivalent  equilibrium  notion  (see  Lemma  2.10  below). 


•  Separately  from  the  literature  on  correlated  equilibria,  the  structure  of  Nash  equilibria 
in  zero-sum  games  with  polynomial  or  separable  (polynomial-like)  utility  functions 
have  been  studied  in  detail  by  Dresher,  Karlin,  and  Shapley.  They  show  how  to 
cast  separable  games  as  finite-dimensional  “convex  games”  by  replacing  the  infinite¬ 
dimensional  mixed  strategy  spaces  with  finite-dimensional  spaces  of  moments  [9]  and 
prove  existence  of  equilibria  via  fixed  point  arguments  [Sj.  There  always  exist  finitely 
supported  equilibria  in  separable  games  as  can  be  shown  using  the  hnite-dimensonality 
of  the  moment  spaces.  The  rich  geometry  of  these  spaces  is  studied  in  [IT].  Most  of 
these  results  as  well  as  ad  hoc  methods  for  computing  equilibria  in  simple  cases  are 
summarized  in  Karlin’s  book  im  The  authors  of  the  present  paper  have  studied 
generalizations  and  extensions  of  these  results  in  nonzero-sum  separable  games 


Second,  our  work  is  related  to  results  from  the  optimization  and  computer  science  liter¬ 
ature. 


•  Aumann  showed  that  the  set  of  correlated  equilibria  of  a  finite  game  is  defined  by 
polynomially  many  (in  the  size  of  the  payoff  tables)  linear  inequalities  |2j .  However,  it 
was  not  clear  whether  this  meant  they  could  be  computed  in  polynomial  time.  This 
question  was  settled  in  the  affirmative  when  Khachian  proved  that  linear  programs 
could  be  solved  in  polynomial  time;  for  an  overview  of  this  and  other  more  efficient 
algorithms,  see  [3]-  Papadimitriou  extended  this  result,  showing  that  correlated  equi¬ 
libria  can  be  computed  efficiently  in  many  classes  of  games  for  which  the  payoffs  can 
be  written  succinctly,  even  if  the  explicit  payoff  tables  would  be  exponential  in  size 

•  The  breakthrough  in  optimization  most  directly  related  to  the  work  in  this  paper  is 
the  development  of  semideffiiite  programming,  a  far-reaching  generalization  of  linear 
programming  which  is  still  polynomial-time  solvable  (for  an  overview,  see  the  appendix 
and  [30]).  More  specifically,  the  development  of  sum  of  squares  methods  has  allowed 
many  optimization  problems  involving  polynomials  or  moments  of  measures  to  be 
solved  efficiently  Parrilo  has  applied  these  techniques  to  efficiently  compute  Nash 
equilibria  of  two-player  zero-sum  polynomial  games  m 

The  remainder  of  this  paper  is  organized  as  follows.  In  Section  [2]  we  define  the  classes 
of  games  we  study  and  correlated  equilibria  thereof,  then  prove  several  characterization 
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theorems.  We  present  algorithms  for  approximating  sample  correlated  equilibria  and  the 
set  of  correlated  equilibria  of  polynomial  games  in  Section  [3j  Then  we  analyze  the  set  of 
correlated  equilibria  of  the  mixed  extension  of  matching  pennies  in  Section  |4|  with  particular 
emphasis  on  computing  extreme  points  of  this  set.  Finally,  we  close  with  conclusions  and 
directions  for  future  work. 


2  Characterizations  of  correlated  equilibria 

In  this  section  we  will  define  finite  and  continuous  games  along  with  correlated  equilibria 
thereof.  We  will  present  several  known  characterizations  of  correlated  equilibria  in  finite 
games  and  show  how  these  naturally  extend  to  continuous  games. 

Some  notational  conventions  used  throughout  are  that  subscripts  refer  to  players,  while 
superscripts  are  frequently  used  for  other  indices  (it  will  be  clear  from  the  context  when  they 
represent  exponents).  If  Sj  are  sets  for  j  =  1 , ,n  then  S  =  ITJ=1Sj  and  ST*  =  Uj^Sj. 
The  n-tuple  s  and  the  (n  —  l)-tuple  s_j  are  formed  from  the  points  s3  similarly.  The  set  of 
regular  Borel  probability  measures  7 r  over  a  compact  Hausdorff  space  S  is  denoted  by  A  (S'). 
For  simplicity  we  will  write  7r(s)  in  place  of  7r({s})  for  the  measure  of  a  singleton  {s}  C  S. 
All  polynomials  will  be  assumed  to  have  real  coefficients. 

2.1  Finite  Games 

We  start  with  the  definition  of  a  finite  game. 

Definition  2.1.  A  finite  game  consists  of  players  i  —  1, . . . ,  n,  each  of  whom  has  a  finite 
pure  strategy  set  Cj  and  a  utility  or  payoff  function  iq  :  C  — >  M,  where  C  =  II ™=1Cj. 

Each  player’s  objective  is  to  maximize  his  (expected)  utility.  We  now  consider  what  it 
would  mean  for  the  players  to  maximize  their  utility  if  their  strategy  choices  were  correlated. 
Let  R  be  a  random  variable  taking  values  in  C  distributed  according  to  some  measure 
Ti  G  A(C').  A  realization  of  A  is  a  pure  strategy  profile  (a  choice  of  pure  strategy  for 
each  player)  and  the  ith  component  of  the  realization  Ri  will  be  called  the  recommendation 
to  player  i.  Given  such  a  recommendation,  player  i  can  use  conditional  probability  to  form 
a  posteriori  beliefs  about  the  recommendations  given  to  the  other  players.  A  distribution 
ti  is  defined  to  be  a  correlated  equilibrium  if  no  player  can  ever  expect  to  unilaterally  gain 
by  deviating  from  his  recommendation,  assuming  the  other  players  play  according  to  their 
recommendations. 

Definition  2.2.  A  correlated  equilibrium  of  a  finite  game  is  a  joint  probability  measure 
n  G  A (C)  such  that  if  R  is  a  random  variable  distributed  according  to  tt  then 

E  [ui(ti,  R-i )  -  Ui(R)\Ri  =  =  ^2  Prob (R  =  s\Ri  =  si)  [ Ui{U ,  S-i)  -  iq(s)]  <  0 

S  —  i(zC—i 

for  all  players  i,  all  s*  G  CJt  such  that  Prob(il;  =  si)  >  0,  and  all  tt  G  C%. 
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While  this  definition  captures  the  idea  we  have  described  above,  the  following  character¬ 
ization  is  easier  to  apply  and  visualize. 

Proposition  2.3.  A  joint  probability  measure  i r  e  A(C)  is  a  correlated  equilibrium  of  a 
finite  game  if  and  only  if 

y,  7r(s)  [ Ui(ti ,  S-i)  -  «i(s)]  <  0  (1) 

for  all  players  i  and  all  Si,U  e  C). 

This  proposition  shows  that  the  set  of  correlated  equilibria  is  defined  by  a  finite  number 
of  linear  equations  and  inequalities  (those  in  (JTj)  along  with  7r(s)  >  0  for  all  s  G  C  and 
SseC7r(s)  =  1)  and  is  therefore  convex  and  even  polyhedral.  It  can  be  shown  via  linear 
programming  duality  that  this  set  is  nonempty  This  can  be  shown  alternatively  by 

appealing  to  the  fact  that  Nash  equilibria  exist  and  are  the  same  as  correlated  equilibria 
which  are  product  distributions. 

We  can  think  of  correlated  equilibria  as  joint  distributions  corresponding  to  recommen¬ 
dations  which  will  be  given  to  the  players  as  part  of  an  extended  game.  The  players  are  then 
free  to  play  any  function  of  their  recommendation  (this  is  called  a  departure  function) 
as  their  strategy  in  the  game.  If  it  is  a  Nash  equilibrium  of  this  extended  game  for  each 
player  to  play  his  recommended  strategy  (i.e.  if  no  player  has  an  incentive  to  unilaterally 
deviate  from  using  the  identity  departure  function),  then  the  distribution  is  a  correlated 
equilibrium.  This  interpretation  is  justified  by  the  following  alternative  characterization  of 
correlated  equilibria. 

Proposition  2.4.  A  joint  probability  measure  tt  e  A(C)  is  a  correlated  equilibrium  of  a 
finite  game  if  and  only  if 


y  7 r(s)  [ui(Ci(s*)>  S-i)  -  itj(s)]  <  0  (2) 

s&C 

for  all  players  i  and  all  functions  Q  :  Cit  C%. 

2.2  Continuous  Games 

Again  we  begin  with  the  definition  of  this  class  of  games. 

Definition  2.5.  A  continuous  game  consists  of  an  arbitrary  (possibly  infinite)  set  /  of 
players  i,  each  of  whom  has  a  pure  strategy  set  Ct  which  is  a  compact  Hausdorff  space  and 
a  utility  function  tq  :  C  — >  M  which  is  continuous. 

Note  that  any  finite  set  forms  a  compact  Hausdorff  space  under  the  discrete  topology 
and  any  function  out  of  such  a  set  is  continuous,  so  the  class  of  continuous  games  includes 
the  finite  games.  Another  class  of  continuous  games  are  the  polynomial  games,  which  are 
our  primary  focus  when  we  study  computation  of  correlated  equilibria  in  the  sections  which 


6 


follow.  The  theorems  and  proofs  below  can  safely  be  read  with  polynomial  games  in  mind, 
ignoring  such  topological  subtleties  as  regularity  of  measures.  However  the  extra  generality  of 
arbitrary  continuous  games  requires  little  additional  work  in  the  proofs  of  the  characterization 
theorems,  so  we  will  not  formally  restrict  our  attention  to  polynomial  games  here. 


Definition  2.6.  A  polynomial  game  is  a  continuous  game  with  n  <  oo  players  in  which  the 
pure  strategy  spaces  are  C*  =  [—1, 1]  for  all  players  and  the  utility  functions  are  polynomials. 


Defining  correlated  equilibria  in  continuous  games  requires  somewhat  more  care  than  in 
finite  games.  The  standard  definition  as  used  in  ra  is  a  straightforward  generalization  of 
the  characterization  of  correlated  equilibria  for  finite  games  in  Proposition  |2.4  In  this  case 
we  must  add  the  additional  assumption  that  the  departure  functions  be  Borel  measurable 
to  ensure  that  the  integrals  are  defined. 


Definition  2.7.  A  correlated  equilibrium  of  a  continuous  game  is  a  joint  probability 

measure  tt  G  A(C)  such  that 


J  s_i)  -  Ui(s)]  dn(s)  <  0 

for  all  i  and  all  Borel  measurable  functions  Q  :  C,  —>  Ct. 


The  problem  of  computing  Nash  equilibria  of  polynomial  games  can  be  formulated  ex¬ 
actly  as  a  finite-dimensional  nonlinear  program  or  as  a  system  of  polynomial  equations  and 
inequalities  The  key  feature  of  the  problem  which  makes  this  possible  is  the  fact  that  it 
has  an  explicit  finite- dimensional  formulation  in  terms  of  the  moments  of  the  players’  mixed 
strategies. 

To  see  this,  suppose  that  player  1  chooses  his  action  x  G  [—1, 1]  according  to  a  mixed 
strategy  a  (a  probability  distribution  over  [—1, 1]).  Each  player’s  utility  function  is  a  mul¬ 
tivariate  polynomial  which  only  contains  terms  whose  degree  in  x  is  at  most  some  constant 
integer  d.  Then  regardless  of  how  everyone  chooses  their  strategies,  their  expected  utility 
will  only  depend  on  a  through  the  moments  J  xda(x),  J  x2da(x), . . . ,  J  xdda(x).  Therefore 
player  1  can  switch  from  a  to  any  other  mixed  strategy  with  the  same  first  d  moments  with¬ 
out  affecting  game  play,  and  we  can  think  of  the  Nash  equilibrium  problem  as  one  in  which 
each  player  seeks  to  choose  moments  which  correspond  to  an  actual  probability  distribution 
and  form  a  Nash  equilibrium. 

On  the  other  hand  there  is  no  exact  finite-dimensional  characterization  of  the  set  of 
correlated  equilibria  in  polynomial  games;  for  a  counterexample  see  Section  [4j  Given  the 
characterization  of  Nash  equilibria  in  terms  of  moments,  a  natural  attempt  would  be  to  try  to 
characterize  correlated  equilibria  in  terms  of  the  joint  moments,  i.e.  the  values  f  s^1  •  •  •  s^n  dn 
for  nonnegative  integers  fc*  and  joint  measures  n.  In  fact  we  will  be  able  to  obtain  such  a 
characterization  below,  albeit  in  terms  of  infinitely  many  joint  moments.  The  reason  this 
attempt  fails  to  yield  a  finite  dimensional  formulation  is  that  the  definition  of  a  correlated 
equilibrium  implicitly  imposes  constraints  on  the  conditional  distributions  of  the  equilibrium 
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measure.  A  finite  set  of  moments  does  not  contain  enough  information  about  these  condi¬ 
tional  distributions  to  check  the  required  constraints  exactly.  Therefore  we  also  consider 
approximate  correlated  equilibria. 

Definition  2.8.  An  e-correlated  equilibrium  of  a  continuous  game  is  a  joint  probability 

measure  7 r  G  A(C')  such  that 

J  [ui(Ci(si),S-i)  -Ui(s)]dir(s)  <  e 

for  all  i  and  all  Borel  measurable  functions  Q  :  C%  — >  C%.  This  definition  reduces  to  that  of 
a  correlated  equilibrium  when  e  =  0. 


Compare  this  definition  to  the  main  characterization  theorem  for  e-correlated  equilibria 
immediately  below.  This  theorem  shows  that  e-correlated  equilibria  can  be  defined  by  inte¬ 
grating  the  utilities  against  any  sufficiently  rich  class  of  test  functions,  instead  of  by  using 
measurable  departure  functions.  While  this  characterization  does  not  have  an  obvious  game 
theoretic  interpretation,  it  allows  us  to  compute  correlated  equilibria  both  algorithmically 
(Section  [3])  and  analytically  (Section  [f]). 

Theorem  2.9.  A  probability  measure  i r  G  A (C)  is  an  e-correlated  equilibrium  of  a  continuous 
game  if  and  only  if  for  all  players  i,  positive  integers  k,  strategies  t},...,t^  G  Ct,  and 
functions  //, . . . ,  ff  :  Q  — >  [0, 1]  in  one  of  the  classes 

1.  Weighted  measurable  characteristic  functions, 

2.  Measurable  simple  functions, 

3.  Measurable  functions, 

4 ■  Continuous  functions, 

5.  Squares  of  polynomials  (if  Ci  C  for  some  kt). 
such  that  Xp=i  fi  ( si )  A  1  for  all  G  Ci,  the  inequality 


holds. 


—  'Uj(s)]  dn  <  e 


(3) 


To  prove  this,  we  need  several  approximation  lemmas. 

Lemma  2.10  (A  special  case  of  Lemma  20  in  [2HJ ) •  Simple  departure  functions  (those  with 
finite  range)  suffice  to  define  e-correlated  equilibria  in  continuous  games.  That  is  to  say,  a 
joint  measure  7 r  is  an  e-correlated  equilibrium  if  and  only  if 

J  [Mi(^(si),s_i)  -  Ui(s)]  dn(s)  <  e 

for  all  players  i  and  all  Borel  measurable  simple  functions  (t  :  CJt  C(. 


Proof.  The  forward  direction  is  trivial.  To  prove  the  reverse,  first  fix  i.  Then  choose  any 
measurable  departure  function  Q  and  let  S  >  0  be  arbitrary.  By  the  continuity  of  tq  and 
compactness  of  the  strategy  spaces  there  exists  a  finite  open  cover  U1,. . .  ,Uk  of  C{  such 
that  Si,  s'  G  U3  implies  |-iq(sj,  s_;)  —  rq(s',  s_;)|  <  5  for  all  s_j  G  and  j  =  1, . . . ,  k.  Fix 
any  si  G  Uk  for  all  j.  Define  a  simple  measurable  departure  function  ft  by  £j(sj)  =  si  where 
j  =  min {7  :  Q(st)  G  U1}.  Then  K(Ci(si)>  «-i)  -  uf^sf),  s_j)|  <  5  for  all  s  G  C,  so 

J  {Ui((i(si),s-i)  -  Ui(s)}dTr(s)  <  J  [Ui(£,i(si),s-i)  +  5  -  Ui(s)]dn(s)  <  e  + 5. 

Letting  5  go  to  zero  completes  the  proof.  □ 

Lemma  2.11.  If  C  is  a  compact  Hausdorff  space,  p,  is  a  finite  regular  Borel  measure  on  C , 
f1, . . . ,  fk  :  C  — >  [0, 1]  are  measurable  functions  such  that  Xp=i  P  <  1;  and  <5  >  0,  then  there 
exist  continuous  functions  g1, . . . ,  gk  :  C  — >  [0, 1]  such  that  g({x  G  C  :  f^{x)  ^  gi(x)})  <  5 
for  all  j  and  Yf)=\  fi3  <  1- 

Proof.  We  can  apply  Lusin’s  theorem  which  states  exactly  this  result  in  the  case  k  =  i  m- 
If  k  >  1,  then  we  can  apply  the  k  —  1  case  with  |  in  place  of  5  to  each  of  the  f3 .  Call 
the  resulting  continuous  functions  g3 .  Then  g({x  G  C  :  f3(x)  ^  g\x)  for  some  j})  <  S. 
But  i  /'  <  !,  so  M(ix  e  C  :  >  1})  <  8.  Let  h(x)  =  max{l,  Yj)=i  93{x)}  so 

h  :  C  — >  [1,  cxd)  is  a  continuous  map.  Dehne  g3(x)  =  Then  the  g3  are  continuous,  sum 

to  at  most  unity,  and  are  equal  to  the  f3  wherever  all  of  the  g3  equal  the  f3,  i.e.  except  on 
a  set  of  measure  at  most  5.  tj 


Lemma  2.12.  If  C  C  is  compact,  f1, . . . ,  fk  :  C  — >  [0, 1]  are  continuous  functions  such 
that  i  / <  I,.  and  8  >  0,  then  there  exist  polynomials  p1, . . .  ,pk  :  C  — >  [0, 1]  which  are 

squares  such  that  \  f3(x)  —  p3(x)  \  <  5  for  all  x  G  C  and  Xq=i7>?  —  1- 

Proof.  By  the  Stone- Weierstrass  theorem,  any  continuous  function  on  a  compact  subset  of 
can  be  approximated  by  a  polynomial  arbitrarily  well  with  respect  to  the  sup  norm. 
Approximating  the  square  root  of  a  nonnegative  function  /  using  this  theorem  and  squaring 
the  resulting  polynomial  shows  that  a  nonnegative  continuous  function  on  a  compact  subset 
of  can  be  approximated  arbitrarily  well  by  a  square  of  a  polynomial  with  respect  to  the 
sup  norm. 

Let  p3  be  a  square  of  a  polynomial  which  approximates  f3  within  A  in  the  sup  norm. 
Let  p3  =  -f-j-  Then  p3  is  always  within  f  of  p3 ,  hence  p3  approximates  f3  within  5  in  the 
sup  norm.  Furthermore  for  all  x  G  C  we  have 


3= 1 


k 


2  j=\ 


p3(  X  )  < 


k 


1+  I  m 

2  j=l 


1. 


□ 


Proof  of  Theorem\2.S\  First  we  prove  that  if  n  is  an  e-correlated  equilibrium  then  (J3])  holds 
in  the  case  where  the  f-  are  simple.  We  can  choose  a  partition  B}, . . . ,  B\  of  Ci  into  disjoint 
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measurable  sets  such  that  //  =  Y^m=i  cjmXBm  where  Cjm  G  [0, 1]  and  XBrn  denotes  the 
indicator  function  which  is  unity  on  B™  and  zero  elsewhere.  Define  a  departure  function 
(^i  :  Ci  — >  Ci  piecewise  on  the  B ™  as  follows.  If 


' B™xC-i 


[ui{tl,S-i)  -  Mj(s)]  dlT 


is  nonnegative  for  some  j  define  Ci(si)  =  f°r  all  s*  G  B ™  where  j  is  chosen  to  maximize 
the  above  integral.  If  the  integral  is  negative  for  all  j  define  Ci(si)  —  si  for  all  s*  G  B™.  Then 
we  have 


1= 1 


'3m 


/  [uiitl,  S-i )  -  Wj(s)]  dn<  [ui(Ci(si),  s-i)  -  rq(s)] 

iB^xC-i  J  By-xC-i 


for  all  m.  Summing  over  m  and  using  the  definition  of  an  e-correlated  equilibrium  yields  ([3]) 
in  the  case  where  the  //  are  simple. 

Conversely  suppose  that  (|3])  holds  for  all  measurable  simple  functions.  Let  Q  :  Ci  — >  Ci 
be  any  simple  departure  function.  Let  be  the  range  of  Q  and  Bj  =  Crl({^})- 

Defining  f-  =  xbj  ,  ((3])  says  exactly  that  7 r  satisfies  the  e-correlated  equilibrium  condition  for 


the  departure  function  Q.  By  Lemma  2.10,  7r  is  an  e-correlated  equilibrium 


Any  simple  function  can  be  written  as  a  sum  of  weighted  characteristic  functions,  so  by 
making  several  of  the  t]  the  same,  we  see  that  (J3])  for  weighted  characteristic  functions  is 
the  same  as  (j3j)  for  simple  measurable  functions.  If  the  inequality  (|3])  holds  for  all  simple 
measurable  functions,  a  standard  limiting  argument  proves  that  it  holds  for  all  measurable 
f\ ,  hence  for  all  continuous  //. 

Suppose  conversely  that  ^  holds  for  all  continuous  //.  Fix  any  measurable  //  sat¬ 
isfying  the  assumptions  of  the  theorem.  Define  a  signed  measure  nj  on  Ci  by  n? (i?*)  = 
Ib  xc  [ui(tJns-i)  ~  Wi(s)]  dn.  Let  /it  =  J2kj=i\7Ti  \  and  fix  any  5  >  0.  Then  by  the  Lemma 
2.11  there  exist  continuous  functions  g\  :  C%  —>  [0, 1]  which  sum  to  at  most  unity  and  equal 
the  f\  except  on  a  set  of  /q  measure  at  most  5.  Therefore 

k  „  k  „ 

d7T~^2  9i  (si)  [ )  -«i(s)]  dvr 
l=i  J  i=i J 

<  \ I  ~  9i(si)\dni  < 


l=i 


so 


dn  <  e  +  2  kS. 


But  5  was  arbitrary,  so  (J3|  holds  for  all  measurable  //. 

Finally  assume  Ci  C  for  some  ki.  If  ([3])  holds  for  all  continuous  //,  then  it  holds  for 
all  squares  of  polynomials.  Suppose  conversely  that  it  holds  for  all  squares  of  polynomials. 
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Let  f-  be  any  continuous  functions  satisfying  the  assumptions  of  the  theorem  and  <5  >  0.  Let 
pi  be  polynomials  squares  which  approximate  the  //  within  6  in  the  sup  norm  and  satisfy 
the  assumptions  of  the  theorem,  as  provided  by  Lemma  2.12  Then 

k  k 

d7T~^  pl(si )  M^-*)  -m*(s)]  d7T 

3= 1 J  3= 1 J 

<  5^  /  \fi(3i)  -  pKsi^dnj  <  SJ2  f  d7rl 

7=1  J  .7  =  1  J 


k  r.  k 

s-i)  -  Mi(s)]  rfvr  <  e  +  5  /  dnl 

3= 1  J  3= 1  J 

But  5  was  arbitrary  and  the  integrals  on  the  right  are  finite,  so  (j3|  holds  for  all  continuous 

n-  □ 


Several  simplifications  occur  when  specializing  Theorem  |2.9|  to  the  e  =  0  case,  yielding 
the  following  characterization.  We  will  use  the  polynomial  condition  of  this  corollary  in 
Section  [3]  to  develop  algorithms  for  computing  (approximate)  correlated  equilibria.  The 
characteristic  function  condition  will  allow  us  to  compute  extreme  correlated  equilibria  of  an 
example  game  in  Section  [4} 


Corollary  2.13.  A  joint  measure  n  is  a  correlated  equilibrium  of  a  continuous  game  if  and 
only  if 

J  fi(si)  [ Ui(ti ,  S-i)  -  -Ui(s)]  d7r(s)  <  0  (4) 

for  all  i  and  ti  G  C)  as  ft  ranges  over  any  of  the  following  sets  of  functions  from  Ci  to  [0,  oo)  : 


1.  Characteristic  functions  of  measurable  sets, 

2.  Measurable  simple  functions, 

3.  Bounded  measurable  functions, 

4 ■  Continuous  functions, 

5.  Squares  of  polynomials  (if  Ci  C  Rki  for  some  ki). 

Proof.  When  e  =  0  the  k  —  1  case  of  equation  (J3])  implies  the  k  >  1  cases.  Furthermore 
e  =  0  makes  (|3])  homogeneous,  so  it  is  unaffected  by  positive  scaling  of  the  //,  which  allows 
us  to  drop  the  assumption  /,  <  1.  □ 


Theorem  2.9  also  has  important  topological  implications  for  the  structure  of  e-correlated 
equilibria.  Recall  that  the  weak*  topology  on  the  set  of  probability  distributions  A(C)  over 
a  compact  Hausdorff  space  is  the  weakest  topology  which  makes  i t  >—*  f  fdix  a  continuous 
functional  whenever  /  :  C  — >  M  is  a  continuous  function. 
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Corollary  2.14.  The  set  of  e- correlated  equilibria  of  a  continuous  game  is  weak*  compact. 


Proof.  By  the  continuous  test  function  condition  in  Theorem  2.9[  the  set  of  e-correlated 
equilibria  is  defined  by  conditions  of  the  form  f  fdn  <  e  where  /  ranges  over  continuous 
functions  of  the  form  fi(si )  S-f)  —  iq(s)].  By  definition  this  presents  the  set  of 

e-correlated  equilibria  as  the  intersection  of  a  family  of  weak*  closed  sets.  Hence  the  set  of  e- 
correlated  equilibria  is  a  closed  subset  of  A(C).  But  A(C)  is  compact  by  the  Banach- Alaoglu 
theorem  [25],  so  the  set  of  e-correlated  equilibria  is  compact.  O 


Corollary  2.15.  If  nk  is  a  sequence  of  ek -correlated  equilibria  and  ek  — ►  0,  then  some  7Tk  is  a 
correlated  equilibrium  or  the  sequence  irk  has  a  limit  point  which  is  a  correlated  equilibrium. 

Proof.  Assume  no  rck  is  a  correlated  equilibrium.  Then  since  ek  — >  0,  the  sequence  nk 
contains  infinitely  many  points.  The  space  A(C)  with  the  weak*  topology  is  compact  by 
the  Banach-Alaoglu  theorem  [25],  hence  any  infinite  set  has  a  limit  point.  Let  n  €  A(C) 
be  a  limit  point  of  the  sequence  nk.  For  any  e  >  0  there  exists  fco  such  that  for  all  k  >  ko, 
7 Tk  is  an  e-correlated  equilibrium.  The  set  A(C')  is  Hausdorff  [25],  so  7r  is  also  a  limit  point 


of  {7T k}k>k0-  Since  the  set  of  e-correlated  equilibria  is  compact  by  Corollary  2.14,  the  limit 
point  7r  must  be  an  e-correlated  equilibrium  for  all  e  >  0,  i.e.  a  correlated  equilibrium.  □ 

Finally,  we  consider  e-correlated  equilibria  which  are  supported  on  some  finite  subset.  In 


this  case,  we  obtain  another  generalization  of  Proposition  2.3 


Theorem  2.16.  A  probability  measure  7 r  £  A(C),  where  C  =  Ilj£iCj  is  a  finite  subset  of  C , 
is  an  e-correlated  equilibrium  of  a  continuous  game  if  and  only  if  there  exist  eiySi  such  that 


£ 


1 r(s)  [Ui(ti,s-i)  -  M;(s)]  <  eMi 


for  all  players  i,  all  s*  £  Ct,  and  all  tt  £  Cl;  and 

eIsi  —  e 

Si^zCi 


for  all  players  i. 

Proof.  If  we  replace  tt  with  in  the  first  inequality  then  sum  over  all  ,sl  £  Cf  and  combine 
with  the  second  inequality,  we  get  the  equivalent  condition  that 

^  tt(s)  K(Ci(s»),  s-i)  -  Ui(s )]  <  e 

sec 


holds  for  all  i  and  any  function  Q  :  Q  — >  (7j.  This  is  exactly  the  definition  of  an  e-correlated 
equilibrium  in  the  case  when  7r  is  supported  on  C.  □ 
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3  Computing  correlated  equilibria 

We  focus  in  this  section  on  developing  algorithms  that  can  compute  approximate  correlated 
equilibria  with  arbitrary  accuracy.  We  consider  three  types  of  algorithms,  which  we  will 
illustrate  in  turn  using  the  example  below. 

Example  3.1.  Consider  the  polynomial  game  with  two  players,  x  and  y,  each  choosing  their 
strategies  from  the  interval  Cx  =  Cy  =  [— 1,1].  Their  utilities  are  given  by 

ux(x,  y )  =  0.596x2  +  2.072 xy  —  0.39 4y2  +  1.360a:  —  1.200 y  +  0.554  and 
uy(x,y )  =  —0.108a:2  +  1.918au/  —  1.044 y2  —  1.232x  +  0.842 y  —  1.886. 

The  coefficients  have  been  selected  at  random.  This  example  is  convenient,  because  as  Figure 
[3]  shows,  the  game  has  a  unique  correlated  equilibrium  (the  players  choose  x  —  y  —  1  with 
probability  one).  For  the  purposes  of  visualization  and  comparison,  we  will  project  the 
computed  equilibria  and  approximations  thereof  into  expected  utility  space,  i.e.  we  will  plot 
pairs  (f  uxd7r,f  uydi r). 

3.1  Static  Discretization  Methods 

The  techniques  in  this  subsection  are  general  enough  to  apply  to  arbitrary  continuous  games 
with  finitely  many  players,  so  we  will  not  restrict  our  attention  to  polynomial  games  here. 
The  basic  idea  of  static  discretization  methods  is  to  select  some  finite  subset  Ci  C  Ci 
of  strategies  for  each  player  and  limit  his  strategy  choice  to  that  set.  Restricting  the  utility 
functions  to  the  product  set  C  =  II”=  x Ct  produces  a  finite  game,  called  a  sampled  game  or 
sampled  version  of  the  original  continuous  game.  The  simplest  computational  approach  is 
then  to  consider  the  set  of  correlated  equilibria  of  this  sampled  game.  This  set  is  defined  by 
the  linear  inequalities  in  Proposition  |2.3|  along  with  the  conditions  that  7r  be  a  probability 
measure  on  C .  The  complexity  of  this  approach  in  practice  depends  on  the  number  of  points 
in  the  discretization. 

The  question  is  then:  what  kind  of  approximation  does  this  technique  yield?  In  general 
the  correlated  equilibria  of  the  sampled  game  may  not  have  any  relation  to  the  set  of  corre¬ 
lated  equilibria  of  the  original  game.  The  sampled  game  could,  for  example,  be  constructed 
by  selecting  a  single  point  from  each  strategy  set,  in  which  case  the  unique  probability  mea¬ 
sure  over  C  is  automatically  a  correlated  equilibrium  of  the  sampled  game  but  is  a  correlated 
equilibrium  of  the  original  game  if  and  only  if  the  points  chosen  form  a  pure  strategy  Nash 
equilibrium.  Nonetheless,  it  seems  intuitively  plausible  that  if  a  large  number  of  points  were 
chosen  such  that  any  point  of  C\  were  near  a  point  of  C,  then  the  set  of  correlated  equilibria 
of  the  finite  game  would  be  “close  to”  the  set  of  correlated  equilibria  of  the  original  game  in 
some  sense,  despite  the  fact  that  each  set  might  contain  points  not  contained  in  the  other. 

To  make  this  precise,  we  will  show  how  to  choose  a  discretization  so  that  the  correlated 
equilibria  of  the  finite  game  are  e-correlated  equilibria  of  the  original  game. 

Proposition  3.2.  Consider  a  continuous  game  with  finitely  many  players,  strategy  sets  Ci, 
and  payoffs  Ui.  For  any  e  >  0,  there  exists  a  finite  open  cover  Uf  ,  Ulf  of  Ci  such  that  if 
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Ci  C  Ci  is  a  finite  set  chosen  to  contain  at  least  one  point  from  each  U\,  then  all  correlated 
equilibria  of  the  finite  game  with  strategy  spaces  Ci  and  utilities  uf\g  will  be  e-correlated 
equilibria  of  the  original  game. 


Proof.  Note  that  the  utilities  are  continuous  functions  on  a  compact  set,  so  for  any  e  >  0 
we  can  choose  a  finite  open  cover  Uf,...,  U1!  such  that  if  st  varies  within  one  of  the  U\  and 
S-i  G  C-i  is  held  fixed,  the  value  of  Ui  changes  by  no  more  than  e.  Let  C  satisfy  the  stated 
assumption  and  let  n  be  any  correlated  equilibrium  of  the  corresponding  finite  game.  Then 


by  Proposition  2.3 


E  7r(s)  s-*) -  - 0 


S  —  iEC—i 


for  all  i  and  all  Sifii  G  Ci.  Any  tt  G  Ci  belongs  to  the  same  U\  as  some  U  G  Ci,  so 


E  vr (s)[ui(U,S-i)  -  ufis)]  <  E  n(s)  [ui(ti,s-i)  -  ufis)  +  e]  <  e  E  n(s)  =  e. 

s-iCiC-i  s—iC:C—i  s-iEC-i 


Therefore  the  assumptions  of  Theorem 


2.16 


are  satisfied  with 


=  eE 


S  —  iEC—i 


vr(s). 


H: 


Combined  with  Corollary  2.15  and  the  fact  that  all  finite  games  have  correlated  equilibria 
this  proposition  shows  that  any  continuous  game  with  finitely  many  players  has  a 
correlated  equilibrium  (the  case  of  an  arbitrary  set  of  players  has  appeared  in  [12])  which 
can  be  computed  as  a  limit  point  of  a  sequence  of  correlated  equilibria  of  sampled  games 
when  the  discretization  gets  finer  and  finer.  The  proof  also  shows  that  if  the  utilities  are 
Lipschitz  functions,  such  as  polynomials,  then  the  U\  can  in  fact  be  chosen  to  be  balls  with 
radius  proportional  to  e.  If  the  strategy  spaces  are  Ci  =  [—1, 1]  as  in  a  polynomial  game, 
then  Ci  can  be  chosen  to  be  uniformly  spaced  within  [—1, 1].  In  this  case  e  =  O  Q)  where 


d  =  max,' 


Ci 


Example  3.1  (continued).  Figure  [I]  is  a  sequence  of  static  discretizations  for  this  game  for 
increasing  values  of  d ,  where  d  is  the  number  of  points  in  Cx  and  Cy.  These  points  are 
selected  by  dividing  [—1, 1]  into  d  subintervals  of  equal  length  and  letting  Cx  =  Cy  be  the 
set  of  midpoints  of  these  subintervals.  For  this  game  it  is  possible  to  show  that  the  rate  of 
convergence  is  in  fact  0  Q)  so  the  worst  case  bound  on  convergence  rate  is  achieved  in  this 
example. 


In  fact  we  can  improve  this  convergence  rate  to  e  =  O  (Jj)  if  we  include  the  endpoints 
±1  in  Ci  as  well  and  assume  that  the  utilities  have  bounded  second  derivatives.  This  fact  is 
based  on  the  following  technical  lemma. 


Lemma  3.3.  Let  x0  >  0.  If  f(±x 0)  <  0  and  ^  >  —M  on  [ — a70,  rr0] ,  then  f(x)  <  for 
all  x  G  [— xo,a;o].  Furthermore  this  bound  is  achieved  by  f(x)  =  (xq  —  x)(a;o  +  x)  at  x  =  0. 


Proof.  Fix  xi  G  argmax^^^  f(x).  Replacing  ffx)  by  /(— x)  if  necessary  we  can  assume 
that  X\  >  0.  Then  =  0,  so  the  function  g(x)  =  f(x i  +  [x|)  satisfies  all  the 
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Static  Discretization  Correlated  Equilibrium  Payoffs  for  d  =  10  to  50  (d  =  strategies  /  player) 
-1.48  r 


-1.5 


-1.52 


-1.54 


-1.56 


-1.58 


Exact  correlated 
equilibrium  payoff 


.  d  =  35  X 
d  =  30  ^4  - 

d  =  25  O 


0  =  0 
$ 


d  =  20 


d  =  15 

* 


d  =  10 

V 


Figure  1:  Convergence  of  a  sequence  of  e-correlated  equilibria  of  the  game  in  Example  |3.1| 
computed  by  a  sequence  of  static  discretizations,  each  with  some  number  d  of  equally  spaced 
strategies  chosen  for  each  player.  The  axes  represent  the  utilities  received  by  players  x  and 
y.  It  can  be  shown  that  the  convergence  in  this  example  happens  at  a  rate  0  Q). 


assumptions  of  the  theorem  (in  particular  it  is  twice  differentiable  despite  the  x  term)  and 
and  achieves  its  maximum  value  of  /(x i)  at  x  —  0. 

2 

Therefore  to  prove  the  lemma  it  suffices  to  prove  that  /( 0)  <  ^  under  the  additional 
assumption  that  )£ (0)  =  0.  By  assumption  >  —M.  Integrating  from  0  to  x,  we  have 
^(x)  >  —Mx.  Integrating  again  from  0  to  x0  yields  /( 0)  —  <  /(x0)  <0.  □ 


Corollary  3.4.  If  f  :  [—1,1]  — >  M  is  a  function  whose  second  derivatives  is  bounded  below 
by  —M  such  that  f  is  nonpositive  at  d  +  1  evenly  spaced  points  from  —1  to  1  inclusive,  then 
max_i<j<i  f(x)  <  j§2- 


Proof.  Apply  Lemma  3.3| between  each  pair  of  adjacent  points  where  /  is  nonpositive.  □ 


The  extremal  example  in  Lemma  3ffi  makes  it  clear  that  we  cannot  achieve  a  tighter 
bound  on  the  values  of  /  by  constraining  higher  derivatives.  Furthermore,  if  we  do  not 
constrain  /  to  be  nonpositive  at  the  endpoints  ±1,  then  we  can  produce  a  linear  /  with 
max_1<3.<1  =  0  (1)  even  if  we  constrain  the  first  derivative  of  /.  This  weaker  convergence 
bound  leads  to  e  =  0  Q)  in  the  example  above. 


Corollary  3.5.  Consider  a  continuous  game  with  strategy  spaces  Ci  =  [—1, 1]  for  all  players 
and  utilities  such  that  Cfft  >  —  M  for  some  M .  Let  Ci  be  the  set  which  consists  of  d  +  1 
points  equally  spaced  between  —1  and  1  inclusive  for  all  i.  Then  a  correlated  equilibrium  of 
the  finite  game  with  strategy  spaces  Ci  and  utilities  is  an  ^ -correlated  equilibrium  of 
the  original  game. 
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Proof.  Let  7 r  be  such  a  correlated  equilibrium.  Then  by  assumption  the  function: 


fi,Si(ti)  =  7r(s)  -s-i)  -  «*(«)] 

is  zero  for  ti  G  Ci  and  has  second  derivative  bounded  below  by  M  Es_ie<7_i  n(,s)-  By  Corollary 

ei)St  =  max_1<t.<1  fi,8i(ti)  ^  ip  E^ec.,  7r(s)-  Therefore  e  =  max,  £s.e6.  eija.  <  ^  and 

□ 


3.4 


t  heorem  2.16  completes  the  proof. 


3.2  Adaptive  Discretization  Methods 

3.2.1  A  family  of  convergent  adaptive  discretization  algorithms 

In  this  section  we  consider  continuous  games  with  finitely  many  players  and  provide  two 
algorithms  (the  second  is  in  fact  a  parametrized  family  of  algorithms  which  generalizes 
the  first)  to  compute  a  sequence  of  efc-correlated  equilibria  such  that  lirn^oc  =  0.  By 
Corollary  |2. 15  any  limit  point  of  this  sequence  is  a  correlated  equilibrium.  We  will  show  that 
for  polynomial  games  these  algorithms  can  be  implemented  efficiently  using  semideffihte 
programming. 

Informally,  these  algorithms  work  as  follows.  Each  iteration  k  begins  with  a  finite  set 
Cf  C  Ci  of  strategies  which  each  player  i  is  allowed  to  play  with  positive  probability  in  that 
iteration;  the  initial  choice  of  this  set  at  iteration  k  =  0  is  arbitrary.  We  then  compute  the 
“best”  e-correlated  equilibrium  in  which  players  are  restricted  to  use  only  these  strategies, 
i.e. ,  the  one  which  minimizes  e  (subject  to  some  extra  technical  conditions  needed  to  ensure 
convergence) . 

Given  the  optimal  objective  value  ek  and  optimal  probability  distribution  nk,  there  is  some 
player  i  who  can  improve  his  payoff  by  ek  if  he  switches  from  his  recommended  strategies  to 
certain  other  strategies.  We  interpret  these  other  strategies  as  good  choices  for  that  player 
to  use  to  help  make  ek  smaller  in  later  iterations  k.  Therefore  we  add  these  strategies  to  Cf 
to  get  Ck+1  and  repeat  this  process  for  iteration  k  +  1. 

Algorithm  3.6.  Fix  a  continuous  game  with  finitely  many  players.  Let  k  —  0  and  for  each 
player  fix  a  finite  subset  Cf  C  Ct. 

•  Let  7 rk  be  an  efc-correlated  equilibrium  of  the  game  having  minimal  ek  subject  to  two 
extra  conditions.  First,  nk  must  be  supported  on  Ck.  Second,  we  require  that  nk  be 
an  exact  correlated  equilibrium  of  the  finite  game  induced  when  deviations  from  the 
recommended  strategies  are  restricted  to  the  set  Ck,  i.e.  when  we  replace  the  condition 


ti  G  Ci  in  Definition  2.8  with  ti  G  Cf. 


That  is  to  say,  let  ek  be  the  optimal  value  of  the  following  optimization  problem,  and 
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7 Tk  be  an  optimal  assignment  to  the  decision  variables. 


minimize  e 
subject  to 


[' Ui(ti,s_i )  -  tq(s)]  <  0 

for  all  i  and  st ,  L;  G  Ck 

s-i&CK 

n(s)[Ui 

(ti i  S—i)  ^z(^)]  ^  £ i,Si 

for  all  i,  Si  G  Ck and  U  G  C% 

S-i&CK 

—  e 

for  all  i 

SiGCk 

7 r(s)  >  0 

for  all  s  G  C 

Y  =  1 

s£Ck 

•  If  ek  —  0,  terminate. 

•  For  each  player  i  for  whom  YlSiec-  eM;  =  e>  f°rm  Ck+1  from  Ck  by  adding  in  at  least 
one  strategy  which  makes  ^2s_.Gc^,  7r(s)  lui(ti,s-i)  —  Wj(s)]  =  for  each  st  e  Ck 
such  that  eitSi  >  0. 

•  For  all  other  players  i,  let  Ck+1  =  Ck. 


•  Let  k  —  k  +  1  and  repeat. 


Note  that  all  steps  of  this  algorithm  are  well-defined.  First,  the  optimization  problem 
is  feasible.  To  see  this  let  7rk  be  any  exact  correlated  equilibrium  of  the  finite  game  with 
strategy  spaces  Ck  and  utilities  tq  restricted  to  Ck\  such  an  equilibrium  exists  because  all 
finite  games  have  correlated  equilibria  m  The  Ui  are  bounded  on  C  (being  continuous 
functions  on  a  compact  set),  so  by  making  e  and  the  large,  we  see  that  nk  is  a  feasible 
solution  of  the  problem.  Second,  the  optimal  objective  value  is  achieved  by  some  nk  because 
the  space  of  probability  measures  on  Ck  is  compact  and  e  is  bounded  below  by  zero.  Third, 
the  set  of  tt  e  C%  making  the  e-correlated  equilibrium  constraints  tight  at  the  optimum  is 
nonempty  by  optimality  of  7rfc  and  continuity  of  ut.  This  set  consists  only  of  strategies  which 
are  not  in  Cf  because  we  have  the  constraint  that  the  deviations  in  utility  be  nonpositive 
for  tt  G  Cf. 


To  show  that  Algorithm  3.6  converges,  we  will  view  it  as  a  member  of  the  following 
family  of  algorithms  with  the  parameters  set  to  a  =  0  and  (3=1.  Varying  these  parameters 
corresponds  to  adding  some  slack  in  the  exact  correlated  equilibrium  constraints  and  allowing 
some  degree  of  suboptimality  in  the  choice  of  strategies  added  to  Ck  to  form  Ck+l .  Such 
changes  make  little  conceptual  difference,  but  could  be  helpful  in  practice  by  making  the 
optimization  problem  strictly  feasible  and  allowing  it  to  be  solved  to  within  a  known  fraction 
of  the  optimal  objective  value  rather  than  all  the  way  to  optimality.  We  will  prove  that  all 
algorithms  in  this  family  converge,  that  is,  with  these  algorithms  ek  converges  to  zero  in  the 
limit. 
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Algorithm  3.7.  Fix  a  continuous  game  with  finitely  many  players  and  parameters  0  <  a  < 
(3  <  1.  Let  k  =  0  and  for  each  player  fix  a  finite  subset  C4°  C  Ct. 

•  Choose  ek  to  be  the  smallest  number  for  which  there  exists  nk  such  that: 

—  7rfc  is  a  probability  distribution  supported  on  Ck , 

—  nk  is  an  efc-correlated  equilibrium  of  the  game, 

—  7ik  is  not  an  e-correlated  equilibrium  for  any  e  <  ek, 

—  7 Tk  is  an  ccefc-correlated  equilibrium  of  the  game  when  strategy  deviations  are 
restricted  to  Ck  (i.e. ,  when  the  condition  ti  E  Ci  is  changed  to  U  E  Ck  in  Definition 

278]) . 

•  If  ek  =  0,  terminate. 

•  For  at  least  one  value  of  i,  form  Ck+l  from  Ck  by  adding  strategies  titSi  E  Ci  such  that 

y  vr k(s)  [ Ui(tijSi,s-i )  -  Ui(s)]  >  f3ek. 

s£Ck 


•  For  all  other  values  of  i,  let  Ck+1  =  Ck. 

•  Let  k  =  k  +  1  and  repeat. 


In  this  case  it  is  not  immediately  obvious  that  the  first  step  of  the  algorithm  is  well- 
defined,  i.e.  that  a  minimal  ek  exists.  To  see  this  note  that  we  could  choose  7rk  to  be  an  exact 
correlated  equilibrium  when  strategy  deviations  are  restricted  to  Ck,  so  there  exists  a  pair 
(7 rM,eM)  satisfying  the  four  conditions  under  the  first  bullet  above.  Choose  some  sequence 
(7ik,l,ek’1),  l  =  1,2, . . .,  of  pairs  satisfying  these  conditions  such  that  ek  =  \imi^00ek'1  is  the 


infimum  over  all  ek  values  of  pairs  satisfying  these  conditions. 


Passing  to  a  subsequence  if 

It  is 


necessary  we  can  assume  without  loss  of  generality  that  the  7ik’1  converge  to  some  7Tk 
clear  from  the  proof  of  Corollary  2.15  that  7 xk  is  an  efc-correlated  equilibrium  supported  on 
Ck  which  is  an  aefc-correlated  equilibrium  when  deviations  are  restricted  to  Ck . 

From  Theorem  2.16  we  see  that  a  small  change  in  the  probabilities  7r(s)  of  a  distribution 
supported  on  a  finite  set  results  in  a  correspondingly  small  change  of  the  minimal  e  for 
which  that  distribution  is  an  e-correlated  equilibrium.  Therefore  nk  is  not  an  e-correlated 
equilibrium  for  any  e  <  ek.  Note  that  this  final  step  depends  crucially  on  the  fact  that  Ck 
is  finite  and  fixed  while  l  varies.  Also  note  that  this  subtlety  disappears  if  a  =  0  because 
in  that  case  it  wouldn’t  matter  if  the  limiting  distribution  had  a  smaller  e  value.  It  is  clear 
that  the  remaining  steps  of  the  algorithm  are  well-defined. 


Theorem  3.8.  Fix  a  continuous  game  with  finitely  many  players, 
converge  in  the  sense  that  ek  — >  0. 


Algorithms  5th  and  3/7 
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Proof.  Suppose  not,  so  there  exists  e  >  0  and  infinitely  many  values  of  k  such  that  ek  >  e.  For 
each  i  let  B}} . . . ,  Blf  be  a  finite  open  cover  of  Ct  such  that  Uifsi,  S-i)  —  Ui(ti ,  S-i )  <  \((3  —  a)e 
when  Si  and  ti  belong  to  the  same  set  B\  and  s_;  G  C-i.  Such  a  cover  exists  by  the 
compactness  of  the  Ci  and  the  continuity  of  the  Ui.  There  are  finitely  many  sets  B\  so  there 
is  some  iteration  k ,  which  we  can  take  to  satisfy  ek  >  e,  such  that  for  all  i  all  of  the  sets  B\ 
which  will  ever  contain  an  element  of  Ck  at  some  iteration  k  already  do. 

Note  that  nk  is  an  aefc-correlated  equilibrium  when  strategy  choices  are  restricted  to  Ck, 
and  ek  >  0  so  we  have  /3ek  >  aek.  By  the  minimality  of  efc,  the  set  Ck+1  \Ck  is  nonempty  for 
some  player  i  (that  is  to  say,  it  is  always  possible  to  perform  the  third  step  of  the  algorithm). 
Choose  such  an  i  and  let  tijSi  G  Ck+1  satisfy 

y:  vr k(s)  [ui(titSi,  s_i)  -  rq(s)]  >  (3ek 

seCk 

for  all  Si  G  Ck.  By  assumption,  for  any  choice  of  ritSi  G  Ck  we  have 

y:  nk(s)  [ui{ri,si)  S-i)  -Ui(s))  <  aek, 

s£Ck 

50  _ 

y  7Tfe(s)  [ui(ti:Si,S-i)  -Ui(ritSi,s-i)]  >  ((3  —  a)ek . 

seCk 

By  construction  of  k,  we  can  choose  r*)Si  G  Ck  to  lie  in  the  same  set  B[  as  ti)Si  for  each 

51  G  Ck.  Thus 


(/ 3  —  a)e  <  {(3  —  a)ek 


< 


^  ^  7T  (s)  \_Ui  (t i,Si ,  S—i)  Ui  {v itSi ,  S—  j)] 

s£Ck 

—  ^  '  7T  (s)  \Ui  (t  i,Si ,  S—i)  Ui(r  i^Si,  S —f)  | 

s£Ck 

<  ^2  nk(o)  ^  ~  °^€  =  ^  ~ 

s£Ck 


a  contradiction. 

Now  we  will  illustrate  Algorithm  |3.6|  on  two  examples. 


□ 


3.6 


initialized  with  =  C9.  = 

d,  y 


Example  3.1  (continued).  In  Figure  [2]  we  illustrate  Algorithm 
{0}.  In  this  case  convergence  is  obtained  in  three  iterations,  significantly  faster  than  the 
static  discretization  method.  The  resulting  strategy  sets  were  Cf  =  Cy  =  { 0,1}. 
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Adaptive  Discretization  e-Correlated  Equilibrium  Payoffs 


-1.5 


-3 


0.5 


1 st  iteration 
e  =  1.956 


1.5 


e  =  4.1 63x  10  " 

(exact  correlated  equilibrium) 


2  iteration 
e  =  1.716 


Figure  2:  Convergence  of  Algorithm  3.6  (note  the  change  in  scale  from  Figure  [I]).  At  each 
iteration,  the  expected  utility  pair  is  plotted  along  with  the  computed  value  of  e  for  which 
that  iterate  is  an  e-correlated  equilibrium  of  the  game.  In  this  case  convergence  to  e  =  0  (to 
within  numerical  error)  occurred  in  three  iterations. 
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k 

ek 

Ckx  \  C T1 

ck  \  C*"1 

ck  \  c t1 

0 

0.99 

{0} 

{0} 

{0} 

1 

4.16 

{0.89} 

2 

5.76 

{-1} 

3 

0.57 

{1} 

4 

0.28 

{0.53} 

{0.50,0.63} 

5 

0.16 

{0.49,0.70} 

6 

10“7 

{-1,0.60} 

{-0.60,0.47} 

Table  2:  Output  of  Algorithm  3.6  on  a  three  player  polynomial  game  with  utilities  of  degree 
4  and  randomly  chosen  coefficients. 


Example  3.9.  For  a  more  complex  illustration,  we  consider  a  polynomial  game  with  three 
players,  choosing  strategies  x,  y,  and  z  G  [—1, 1].  The  utilities  were  chosen  to  be  polynomials 
with  terms  np  to  degree  4  in  all  the  variables  and  the  coefficients  were  chosen  independently 
according  to  a  normal  distribution  with  zero  mean  and  unit  variance  (their  actual  values  are 
omitted).  Algorithm  3.6  proceeds  as  in  Table  [2J  which  shows  the  value  of  ek  and  the  new 
strategies  added  on  each  iteration.  The  terminal  probability  distribution  7 r6  does  not  display 
any  obvious  structure;  in  particular  it  is  not  a  Nash  equilibrium  (product  distribution). 


3.2.2  Implementing  these  algorithms  with  semidefinite  programs 

To  implement  these  algorithms  for  polynomial  games,  we  must  be  able  to  do  two  things. 
First,  we  need  to  solve  optimization  problems  with  finitely  many  decision  variables,  linear 
objective  functions  and  two  types  of  constraints:  nonnegativity  constraints  on  linear  func¬ 
tionals  of  the  decision  variables,  and  nonnegativity  constraints  on  univariate  polynomials 
whose  coefficients  are  linear  functionals  of  the  decision  variables.  That  is  to  say,  we  must  be 
able  to  handle  constraints  of  the  form  p(t)  >  0  for  all  t  G  [—1, 1],  where  the  coefficients  of 
the  polynomial  p  are  linear  in  the  decision  variables.  Second,  we  need  to  extract  values  of  t 
for  which  such  polynomial  inequalities  are  tight  at  the  optimum. 

Both  of  these  tasks  can  be  done  simultaneously  by  casting  the  problem  as  a  semidefinite 
program  (SDP).  For  an  overview  of  semidefinite  programs  and  a  summary  of  the  necessary 
results  (both  of  which  are  classical),  see  the  appendix. 


In  the  optimization  problem  in  Algorithm  3.6  we  have  a  finite  number  of  univariate 


polynomials  in  ti  whose  coefficients  are  linear  in  the  decision  variables  7r(s)  and  6jjSi.  We 
wish  to  constrain  these  coefficients  to  allow  only  polynomials  which  are  nonnegative  for  all 
ti  G  [—1, 1].  By  Propositions  A. 3  and  A. 4  in  the  appendix  this  is  the  same  as  asking  that 
these  coefficients  equal  certain  linear  functions  of  matrices  (i.e. ,  sums  along  antidiagonals) 
which  are  constrained  to  be  symmetric  and  positive  semidefinite.  Therefore  we  can  write 
this  optimization  problem  as  a  semidefinite  program. 

As  a  special  case  of  convex  programs,  semidefinite  programs  have  a  rich  duality  theory 
which  is  useful  for  theoretical  and  computational  purposes.  In  particular,  SDP  solvers  keep 
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3.7 


a 

b 

c 

a 

0 

1 

0 

b 

1 

5 

7 

c 

0 

7 

0 

Tabic  3:  A  finite  symmetric  game  with  identical  utilities  for  which  Algorithm 
a  —  (3  —  1  does  not  converge  when  started  with  strategy  sets  C®  =  C%  —  {«}• 


with 


track  of  feasible  primal  and  dual  solutions  in  order  to  determine  when  optimality  is  reached. 
It  can  be  shown  that  the  dual  data  obtained  by  an  SDP  solver  run  on  this  optimization 
problem  will  encode  the  values  of  tl  making  the  polynomial  inequalities  tight  at  the  optimum 

H3|. 

The  process  of  generating  an  SDP  from  the  optimization  problem  in  the  algorithms  above, 
solving  it,  and  extracting  an  optimal  solution  along  with  tt  values  from  the  dual  can  all  be 
automated.  We  have  done  so  using  the  SOSTOOLS  MATLAB  toolbox  for  the  pre-  and 
post-processing  and  SeDuMi  for  solving  the  semidefinite  programs  efficiently  [22l  129], 


3.2.3  A  nonconvergent  limiting  case 


Note  that  in  the  algorithms  above  the  convergence  of  the  sequence  ek  is  not  necessarily 
monotone.  If  we  were  to  let  a  —  0  (a  case  we  did  not  allow  above),  the  sequence  would 
become  monotone  nonincreasing.  If  we  were  to  furthermore  fix  a  —  (3  —  1,  then  the  condition 
that  7 r  be  an  exact  (or  aek -)  correlated  equilibrium  when  deviations  are  restricted  to  Ck  would 
become  redundant  and  could  be  removed. 


These  changes  would  simplify  the  behavior  of  Algorithm  3.7  conceptually  as  well  as 
reducing  the  size  of  the  SDP  solved  at  each  iteration,  so  we  would  like  to  adopt  them  if 
possible.  However,  the  resulting  algorithm  may  not  converge,  in  the  sense  that  ek  may 
remain  bounded  away  from  zero. 

Example  3.10.  Consider  the  game  shown  in  Table  |3j  which  is  symmetric  and  has  identical 
utilities  for  both  players.  Let  C®  =  C®  =  {a}  and  apply  Algorithm  3.6,  but  remove  the 
condition  that  nk  be  an  exact  correlated  equilibrium  when  deviations  are  restricted  to  Ck. 
The  only  probability  distribution  supported  on  C°  is  which  has  an  objective  value  of 
e°  =  1.  It  is  easy  to  see  that  C}  is  formed  by  simply  adding  each  player’s  best  response  to 
a,  so  that  C\  =  C\  =  {a,  b}.  We  will  argue  that  the  unique  solution  to  the  optimization 
problem  in  iteration  k  —  1  is  also  <5(a,a),  hence  Cf  =  C\  and  the  algorithm  gets  “stuck”,  so 
that  ek  =  e°  =  1  for  all  k. 

For  a  probability  distribution  7r,  let  nT  denote  n  with  the  players  interchanged.  By 
symmetry  and  convexity,  if  n  is  an  optimal  solution  then  so  is  7r+27r  ,  which  is  a  symmetric 
probability  distribution  with  respect  to  the  two  players.  Hence  an  optimal  solution  which 
is  symmetric  always  exists.  We  will  parametrize  such  distributions  by  7r  =  p5(a,a)  +  q$(a,b)  + 
q5(bta)  +  r5(b,b),  where  p,  q,  r  >  0  and  p  +  2q  +  r  =  1.  Define  a  departure  function  (  :  C\  — >  C\ 
by  C(a)  —  b,  ((b)  =  ((c)  =  c.  Then  for  7r  to  be  an  e-correlated  equilibrium  it  must  satisfy 
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k 

ek 

ckx  \  cr 1 

Ckv  \  ck~l 

0 

2 

{-i} 

{-1} 

1 

4 

{0} 

{0} 

2 

0 

{i} 

{1} 

Table  4: 
a  =  f3  = 


Output  of  Algorithm  3.6  for  a  polynomial  game  on  which  Algorithm  |3.7  with 
1  does  not  converge  to  a  correlated  equilibrium. 


the  following  condition: 

e  >  7r(s)  MC(si)>  s2)  -  wi(si,  s2)] 

siecf  sec1 

=  p  +  Aq  —  q  +  2r  =  1  +  q  +  r. 


We  know  we  can  achieve  e  =  1  with  p  —  1  (i.e.  n  =  n0  =  <5(a,a));  and  this  inequality 
shows  that  if  p  <  1  then  e  >  1.  Therefore  the  minimal  e  value  in  iteration  k  —  1  is 
unity  and  is  achieved  by  n  =  5(a,a)-  Furthermore  we  have  shown  that  this  is  the  unique 
symmetric  probability  distribution  which  achieves  the  minimal  value  of  e.  Hence  any  other 
(not  necessarily  symmetric)  optimal  solution  i f  satisfies  7r+97r  =  <5(a,a)-  But  S^a)  is  an  extreme 
point  of  the  convex  set  of  probability  distributions  on  C1,  so  we  must  in  fact  have  n  =  <5(a,a)- 
Therefore  nl  =  7T°  =  <5(a,a)  is  the  unique  optimal  solution  on  iteration  k  —  1,  so  the  procedure 
must  get  stuck  as  claimed.  That  is,  Ck  =  {a,  b}  and  ek  =  1  for  all  k  >  1. 

The  same  behavior  can  occur  in  polynomial  games,  as  can  be  shown  by  “embedding”  the 
above  finite  game  in  a  polynomial  game.  For  example,  we  can  take  Cx  =  Cy  =  [—1, 1]  and 

ux(x,  y )  =  uy(x,  y)  =  (1  -  x2)(3y2  +  6y  +  5) 

+  (1  -y2)  {3x2  +  6x  +  5). 


Then  if  C°  =  C®  =  {—1}  the  same  analysis  as  above  shows  that  Ck  =  Ck  =  {  —  1,0}  and 
ek  =  2  for  all  k  >  1. 


Example  3.11.  If  we  run  Algorithm  3.6|  on  this  polynomial  game,  the  iterations  proceed  as 
in  Table  |4j  The  correlated  equilibrium  obtained  in  iteration  2  is 


7 r2  =  0.4922A  (a:  =  0,  y  =  1)  +  0.4922<5(a:  =  1,  y  =  0) 
+  0.0156h(x  =  1  ,y  =  1), 


i.e.,  a  probability  of  0.4922  is  assigned  to  each  of  the  outcomes  (x,y)  =  (0, 1)  and  (x,y)  = 
(1,0)  and  a  probability  of  0.0156  is  assigned  to  (x,y)  =  (1, 1). 


3.3  Moment  Relaxation  Methods 

In  this  subsection  we  again  consider  only  polynomial  games.  The  moment  relaxation 
methods  for  computing  correlated  equilibria  have  a  different  flavor  from  the  discretization 
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methods  discussed  above.  Instead  of  using  tractable  finite  approximations  of  the  correlated 
equilibrium  problem  derived  via  discretizations,  we  begin  with  the  alternative  exact  char¬ 
acterization  given  in  condition  [5]  of  Corollary  2.13 
correlated  equilibrium  if  and  only  if 


In  particular,  a  measure  tt  on  C  is  a 


P2(Si 


[' Ui(ti ,  s_j)  -  M;(s)]  cLtt(s)  <  0 


(5) 


for  all  i,  ti  G  Ci,  and  polynomials  p.  If  we  wish  to  check  all  these  conditions  for  polynomials 
p  of  degree  less  than  or  equal  to  d,  we  can  form  the  matrices 


Sf  = 


02 


S  i 

s2 

Qd 
^ i 

„2 

„3 

Qd-\- 1 

Si 

si 

•  •  •  si 

s* 

si 

.  .  .  Qd-\~2 

d-\- 1 

Qd-\-  2 

QZd 

Let  c  be  a  column  vector  of  length  d  +  1  whose  entries  are  the  coefficients  of  p,  so  p2(si)  = 
d  Sfc.  If  we  define 

M?{U)  =  j  Sf  [ui(ti,  s_j)  -Ui(s)]dvr(s), 

then  ®i  is  satisfied  for  all  p  of  degree  at  most  d  if  and  only  if  dMf(tj)c  <  0  for  all  c  G  Rd+1 
and  ti  G  Ci ,  i.e.  if  and  only  if  Mf{ti)  is  negative  semidehnite  for  all  ti  G  Ci. 

The  matrix  Mf(U )  has  entries  which  are  polynomials  in  t,j  with  coefficients  which  are 
linear  in  the  joint  moments  of  tt.  To  check  the  condition  that  Mf(tt)  be  negative  semidehnite 
for  all  ti  G  [—1, 1]  for  a  given  d  we  can  use  a  semidehnite  program  (Proposition  A. 5  in  the 


appendix),  so  as  d  increases  we  obtain  a  sequence  of  semidehnite  relaxations  of  the  correlated 
equilibrium  problem  and  these  converge  to  the  exact  condition  for  a  correlated  equilibrium. 

We  can  also  let  the  measure  tt  vary  by  replacing  the  moments  of  n  with  variables  and 
constraining  these  variables  to  satisfy  some  necessary  conditions  for  the  moments  of  a  joint 
measure  on  C  (see  appendix).  These  conditions  can  be  expressed  in  terms  of  semidehnite 
conditions  and  there  is  a  sequence  of  these  conditions  which  converges  to  a  description 
of  the  exact  set  of  moments  of  a  joint  measure  tt.  Thus  we  obtain  a  nested  sequence  of 
semidehnite  relaxations  of  the  set  of  moments  of  measures  which  are  correlated  equilibria, 
and  this  sequence  converges  to  the  set  of  correlated  equilibria. 

Example  3.1  (continued).  Figure  [3]  shows  moment  relaxations  of  orders  d  =  0, 1,  and  2. 
Since  moment  relaxations  are  outer  approximations  of  the  set  of  a  correlated  equilibria  and 
the  2nd  order  moment  relaxation  corresponds  to  a  unique  point  in  expected  utility  space,  all 
correlated  equilibria  of  the  example  game  have  exactly  this  expected  utility.  In  fact,  the  set 
of  points  in  this  relaxation  is  a  singleton  (even  before  being  projected  into  utility  space),  so 
this  proves  that  this  example  game  has  a  unique  correlated  equilibrium. 
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Moment  Relaxation  Correlated  Equilibrium  Payoffs  of  Order  d  =  0,  1 ,  and  2 


0.5  1  1.5  2  2.5  3 


Figure  3:  Semidefinite  relaxations  approximating  the  set  of  correlated  equilibrium  payoffs. 
The  second  order  relaxation  is  a  singleton,  so  this  game  has  a  unique  correlated  equilibrium 
payoff  (and  in  fact  a  unique  correlated  equilibrium). 

4  Structure  of  the  set  of  correlated  equilibria 

We  will  demonstrate  the  complexity  of  the  extreme  points  of  the  set  of  correlated  equilibria 
through  several  example  games.  In  particular  we  will  show  that  there  can  be  a  superexpo¬ 
nential  separation  between  the  number  of  extreme  Nash  and  extreme  correlated  equilibria 
of  a  finite  game,  and  that  polynomial  games  can  have  extreme  correlated  equilibria  which 
are  not  finitely  supported.  This  implies  that  in  general  the  condition  that  a  measure  p  be  a 
correlated  equilibrium  of  a  given  polynomial  game  cannot  be  defined  in  terms  of  conditions 
on  finitely  many  joint  moments  of  /i.  In  contrast,  the  condition  that  an  n-tuple  of  mixed 
strategies  be  a  Nash  equilibrium  of  a  polynomial  game  can  always  be  expressed  in  terms  of 
finitely  many  moments  EH- 

First  we  fix  notation.  Given  a  space  S  the  set  A (5)  will  denote  the  set  of  Borel  probability 
measures  on  S  (as  above)  and  the  set  A*  (S')  will  denote  the  set  of  finite  positive  measures 
on  S.  In  particular  A(S)  is  the  set  of  measures  in  A*  (S')  with  unit  mass.  If  S  is  finite  then 
A(S)  is  a  simplex  and  A*(S)  is  an  orthant  in  For  any  p  e  S,  define  5P  e  A(S)  to  be 
the  measure  which  assigns  unit  mass  to  the  point  p.  Let  I  =  [—1, 1]  C  M. 

We  will  focus  on  two  related  examples,  one  with  finite  strategy  sets  and  one  with  infinite 
strategy  sets.  We  will  develop  them  in  parallel  by  analyzing  arbitrary  games  satisfying  the 
following  condition. 

Assumption  4.1.  The  game  is  a  zero-sum  continuous  game  with  two  players,  called  x  and 
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Table  5:  Utilities  for  matching  pennies 


{Ux")  'U'y) 

x  =  —  1  X  =  1 

y  =  - 1 
y  =  1 

(1,-1)  (-1,1) 
(-1,1)  (1,-1) 

y.  The  strategy  sets  Cx  and  Cy  are  compact  subsets  of  /  =  [—1,1],  each  of  which  contains 
a  positive  element  and  a  negative  element.  Player  x  chooses  a  strategy  x  G  Cx  and  player  y 
chooses  y  G  Cy.  The  utility  functions  are  ux(x,y)  =  xy  =  —uy(x,y). 

Example  4.2.  Fix  an  integer  n  >  0.  Let  Cx  and  Cy  each  have  2 n  elements,  n  of  which  are 
positive  and  n  of  which  are  negative.  If  we  take  n  —  1  and  Cx  =  Cy  =  {—1,1}  then  we 
recover  the  matching  pennies  garnej}]  as  shown  in  Table  |5} 

Example  4.3.  Let  Cx  =  Cy  =  [—1, 1].  Then  the  game  is  essentially  the  mixed  extension  of 
matching  pennies.  That  is  to  say,  suppose  two  players  play  matching  pennies  and  choose 
their  strategies  independently,  playing  1  with  probabilities  p  G  [0,1]  and  q  G  [0,1].  Define 
the  utilities  for  the  mixed  extension  to  be  the  expected  utilities  under  this  random  choice 
of  strategies.  Letting  x  =  2p  —  1  and  y  =  2q  —  1,  the  utility  to  the  first  player  is  xy  and 
the  utility  to  the  second  player  is  —  xy.  Therefore  this  example  is  the  mixed  extension  of 
matching  pennies,  up  to  an  affine  scaling  of  the  strategies. 


4.1  Extreme  Nash  equilibria 


We  will  now  show  that  the  extreme  points  of  the  sets  of  Nash  equilibria  in  games  satisfying 
Assumption  4T  are  well-behaved.  Since  these  games  are  zero-sum,  the  set  of  Nash  equilibria 
can  be  viewed  as  a  Cartesian  product  of  two  (weak*)  compact  convex  sets,  the  sets  of 
maximin  and  minimax  strategies.  By  the  Krein-Milman  theorem,  such  sets  can  be  completely 
characterized  in  terms  of  their  extreme  points  [25]. 

We  define  Nash  equilibria  in  two-player  games,  which  will  be  sufficient  for  our  purposes, 
as  well  as  the  standard  notions  of  extreme  point  and  extreme  ray  from  convex  analysis. 


Definition  4.4.  A  Nash  equilibrium  is  a  pair  (a,  r)  G  A (Cx)  x  A (Cy)  such  that  ux(x,  r)  < 
ux(cr ,  r)  for  all  x  G  Cx  and  uy(cr ,  y)  <  uy(c r,  r)  for  all  y  G  Cy. 

Definition  4.5.  A  point  x  in  a  convex  set  K  is  an  extreme  point  if  x  =  Xy  +  (1  —  A )z  for 
y,  z  G  K  and  A  G  [0, 1]  implies  x  =  y  =  z. 

Definition  4.6.  A  convex  set  K  such  that  x  G  K  and  A  >  0  implies  Xx  G  K  is  called  a 
convex  cone.  A  point  x  ^  0  is  an  extreme  ray  of  the  convex  cone  K  if  x  =  y  +  z  and 
y,  z  G  K  implies  that  y  or  z  is  a  scalar  multiple  of  x. 

inspection  of  the  utilities  we  can  see  that  for  any  Cx  and  Cv  with  at  least  two  points,  the  rank  of 
this  game  in  the  sense  of  [27]  is  (1, 1)  (and  in  fact  also  in  the  stronger  sense  of  Theorem  3.3  of  that  paper). 
The  notion  of  the  rank  of  a  game  is  related  to  the  rank  of  the  payoff  matrices  and  will  not  play  a  significant 
role  in  this  paper;  we  merely  wish  to  note  that  under  this  definition  of  complexity  of  payoffs  the  games  we 
consider  are  extremely  simple. 
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The  Nash  equilibria  of  games  satisfying  Assumption  4.1  take  the  following  particularly 
simple  form. 


Proposition  4.7.  A  pair  (a,  r)  G  A (Cx)  x  A (Cy)  is  a  Nash  equilibrium  of  a  game  satisfying 


Assumption  f.l  if  and  only  if  f  xda(x)  =  f  ydr(y )  =  0. 


Proof.  If  f  xdafx )  =  0  then  uy(c r,  y)  —  0  for  all  y  G  Cy,  so  any  r  G  A (Cy)  is  a  best  response  to 
cr.  If  f  ydr(y )  =  0  as  well  then  cr  is  also  a  best  response  to  r,  so  (cr,  r)  is  a  Nash  equilibrium. 

Suppose  for  a  contradiction  that  there  exists  a  Nash  equilibrium  (a,  r)  such  that  J  xda(x)  > 
0;  the  other  cases  are  similar.  Player  y  must  play  a  best  response,  so  J  ydr(y)  <  0,  which 
is  possible  by  assumption.  Player  x  plays  a  best  response  to  that,  so  J  xda(x)  <  0,  a 
contradiction.  □ 


We  introduce  the  notion  of  extreme  Nash  equilibrium  in  the  context  of  zero-sum  games. 
For  an  extension  of  this  definition  to  two-player  non-zero  sum  finite  games  and  a  proof  that 
extreme  Nash  equilibria  are  always  extreme  points  of  the  set  of  correlated  equilibria  in  this 
setting,  see  0  or  [TO]. 


Definition  4.8.  A  Nash  equilibrium  (cr,  r)  G  A(CX)  x  A {Cy)  is  extreme  if  a  and  r 
are  extreme  points  of  the  maximin  ({a  G  A(C'a,)|  f  xda(x)  =  0})  and  minimax  ({r  G 
A(Cy)|  f  ydr(y)  =  0})  sets,  respectively. 

Proposition  4.9.  Consider  a  game  satisfying  Assumption  f.l.  A  pair  (a,  r)  G  A  (Cl 


x 


A (Cy)  is  an  extreme  Nash  equilibrium  if  and  only  if  a  and  r  are  each  either  do  or  of  the 
form  aSu  +  (38v  where  u  <  0,  v,  a,  (3  >  0,  a  +  (3  =  1,  and  au  +  j3v  =  0. 


Proof.  By  Proposition  4/7  we  must  show  that  these  distributions  are  the  extreme  points  of 
the  set  of  probability  distributions  having  zero  mean.  Since  ho  is  an  extreme  point  of  the 
set  of  probability  distributions,  it  must  be  an  extreme  point  of  the  subset  which  has  zero 
mean.  To  see  that  aSu  +  /38v  is  also  an  extreme  point,  suppose  we  could  write  it  as  a  convex 
combination  of  two  other  probability  distributions  with  zero  mean.  The  condition  that  both 
be  positive  measures  implies  that  both  must  be  of  the  form  a'8u  +  f3'8v.  But  a  and  (3  as 
specified  above  are  the  unique  coefficients  which  make  this  be  a  probability  measure  with 
zero  mean.  Therefore  a'  —  a  and  (3'  =  f3 ,  so  a8u  +  (38v  cannot  be  written  as  a  nontrivial 
convex  combination  of  probability  distributions  with  zero  mean,  so  it  is  an  extreme  point. 

Suppose  a  were  an  extreme  point  which  was  not  of  one  of  these  types.  Then  a  could 
not  be  supported  on  one  or  two  points,  so  either  [0, 1]  or  [—1,0)  could  be  partitioned  into 
two  sets  of  positive  measure.  We  will  only  treat  the  first  case;  the  second  is  similar.  Let 
[0, 1]  =  A  U  B  where  A  fl  B  —  0  and  cr(A),  c(B)  >  0.  Since  a  has  zero  mean  we  must  have 
cr([— 1,0))  >  0  as  well. 

For  a  set  D  we  define  the  restriction  measure  a \d  by  u\jj{C)  =  cr(D  fl  C )  for  all  C. 
Then  cr  =  cr\A  +  a\B  +  cr | [_1;0) -  Let  a  =  fAxda(x),  b  =  fBxdcr(x),  and  c  =  J^_10^xda(x). 
Since  cr([— 1,  0))  >  0  and  x  is  less  than  zero  everywhere  on  [—1,  0),  we  must  have  c  <  0.  By 
assumption  a  +  b  +  c  =  0.  Therefore  we  can  write: 


—  (  cr|^4  +  — cr|[_lj0) 


+ 


ct\b  +  -j — per  r—i  o) 
c 
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Being  an  extreme  point  of  the  set  of  probability  measures  with  zero  mean,  a  must  be  an 
extreme  ray  of  the  set  of  positive  measures  with  first  moment  equal  to  zero.  But  this  means 
that  we  cannot  write  a  =  cq  +  cq  where  the  a,  are  positive  measures  with  zero  first  moment 
unless  (jj  is  a  multiple  of  a.  Neither  of  the  measures  in  parentheses  above  is  a  multiple  of  a, 
so  we  have  a  contradiction.  □ 


We  illustrate  this  proposition  on  both  examples  introduced  at  the  beginning  of  this 
section. 


Example  4.2  (cont’d).  In  this  case  neither  Cx  nor  Cy  contains  zero,  so  the  only  extreme  Nash 
equilibria  are  those  in  which  a  and  r  are  of  the  form  a8u  +  (35v  for  u  <  0  and  v  >  0.  For 


any  choice  of  u  and  v  there  are  unique  a  and  f3  satisfying  the  conditions  of  Proposition  4.9 


There  are  n  possible  choices  for  each  of  u  and  v  for  each  of  the  two  players,  so  there  are  n4 
extreme  Nash  equilibria. 


Example  4.3  (cont’d).  Since  Cx  =  Cy  =  [—1,1],  there  are  infinitely  many  extreme  Nash 
equilibria  in  this  case.  However,  they  are  all  finitely  supported  and  the  size  of  their  support 
is  always  either  one  or  two.  Furthermore  the  condition  that  (a,  r)  be  a  Nash  equilibrium 
is  equivalent  to  both  having  zero  mean.  This  illustrates  the  general  facts  that  in  games 
with  polynomial  utility  functions  the  Nash  equilibrium  conditions  only  involve  finitely  many 
moments  of  cr  and  r  and  the  extreme  Nash  equilibria  (when  defined,  i.e. ,  for  zero-sum  games) 
have  uniformly  bounded  support. 


4.2  Extreme  correlated  equilibria 

We  will  show  that  even  in  finite  games,  the  number  of  extreme  correlated  equilibria  can  be 
many  more  than  the  number  of  extreme  Nash  equilibria.  It  makes  sense  to  compare  these 
because  all  extreme  Nash  equilibria  of  a  two-player  finite  game  are  automatically  extreme 
correlated  equilibria  pH]-  Also,  for  rank  (1, 1)  finite  games  we  will  show  that  the  size  of  the 
support  of  extreme  correlated  equilibria  can  be  arbitrarily  large. 

In  the  case  of  polynomial  games  we  will  show  that  even  in  the  rank  (1, 1)  case,  there 
can  be  extreme  correlated  equilibria  with  arbitrarily  large  finite  support  and  without  finite 
support.  This  implies  that  the  set  of  correlated  equilibria  cannot  be  characterized  in  terms 
of  finitely  many  joint  moments. 

Note  that  all  of  the  characterizations  of  exact  correlated  equilibria  in  Section  [2]  are  in 
terms  of  homogeneous  (that  is,  invariant  under  positive  scaling)  linear  constraints  on  p. 
The  only  requirement  on  p  that  is  not  homogeneous  is  the  probability  measure  condition 
p(I  x  I)  =  1.  It  will  be  convenient  to  consider  measures  p  G  A *{CX  x  Cy)  satisfying  all  the 
correlated  equilibrium  conditions  except  for  this  normalization. 


Definition  4.10.  We  refer  to  a  measure  p  G  A (Cx  x  Cy )  satisfying  the  conditions  of  Defini¬ 
tion  2.7  as  a  proper  correlated  equilibrium  and  a  measure  p  G  A *{CX  x  Cy)  satisfying  all 
the  conditions  except  normalization  as  a  homogeneous  correlated  equilibrium.  When 
the  distinction  is  irrelevant  or  clear  from  context  we  simply  use  the  term  correlated  equi¬ 
librium  to  refer  to  either.  In  the  context  of  homogeneous  correlated  equilibria  the  term 


extreme  will  refer  to  extreme  rays;  for  proper  correlated  equilibria  it  will  refer  to  extreme 
points. 

When  fi  ^  0  is  a  homogenous  correlated  equilibrium,  is  a  ProPer  correlated 

equilibrium.  The  set  of  homogenous  correlated  equilibria  is  a  convex  cone.  The  extreme 
rays  of  this  cone  are  exactly  those  measures  which  are  positive  multiples  of  the  extreme 
points  of  the  set  of  proper  correlated  equilibria. 

The  following  proposition  characterizes  correlated  equilibria  of  games  satisfying  Assump¬ 


tion  4.1  and  is  analogous  to  Proposition  4.7  for  Nash  equilibria.  Note  how  the  Nash  equi¬ 
librium  measures  were  characterized  in  terms  of  their  moments  but  the  correlated  equilibria 
are  not. 


Proposition  4.11.  For  a  game  satisfying  Assumption  f.l  and  a  measure  /i  G  A*(CX  x  Cy), 
the  following  are  equivalent: 

1.  y  is  a  correlated  equilibrium; 

2.  If  dn  =  xydy  then  the  marginals 


3. 


kx(A)  :=  /  xyd/j(x,y)  and  ny(A)  :  =  /  xydp(x,y) 

J Axl  JlxA 

are  both  the  zero  measure,  i.e.,  equal  zero  for  all  measurable  A  C  I; 


A* (A)  :=  /  ydy(x,y)  and  X y(A)  :=  /  xdy(x,y) 


'Axl 


I  IxA 


are  both  the  zero  measure. 


Proof.  0AH  We  will  consider  only  kx;  ny  is  similar.  The  characteristic  function 
of  Corollary  2.13  with  /*  =  xi  implies  that 


version 


x'  ydn{x,y)  <  /  xydy(x,y)  <  y'  xdp(x,y) 

Jlxl  Jlxl  Jlxl 

for  all  x'  G  Cx,y'  G  Cy.  By  assumption  it  is  possible  to  choose  x'  and  y'  either  positive  or 
negative,  so  Jf  7  xydy(x,  y)  =  0.  Furthermore  the  same  argument  with  any  A  implies  that 
Sax  j  xydfi{x ,  y)  >  0.  Therefore  we  have 


0  =  /  xyd/ifx,  y)  =  /  xydy(x,  y)  +  /  xydy(x,y)  >  0  +  0  =  0 
Jlxl  J  Axl  J  (I\A)xI 

for  all  A,  so  the  inequality  must  be  tight  and  we  get  fAxI  xydp,(x,  y)  =  0  for  all  A. 

( 2]  [3])  Substituting  this  equation  into  the  characteristic  function  version  of  Corollary 


2.13 


yields  x'  fAxI  ydy(x,  y)  <  0  for  all  x'  G  Cx  and  all  measurable  A.  But  we  can  choose 
x'  to  be  positive  or  negative  by  assumption,  so  we  must  have  fAxIydy(x,y)  =  0  for  all 
measurable  AC/. 
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fh0  Suppose  we  know  that  Ax  is  the  zero  measure.  We  can  approximate  xy  on  /  x  / 
by  functions  of  the  form  YAk= i  xiyXAi(x),  where  the  measurable  sets  A,  have  small  diameter 
and  partition  /,  Xi  G  Ai,  and  xa{  is  the  indicator  function  for  Ai.  By  definition  of  the 
Lebesgue  integral  and  the  dominated  convergence  theorem  m  we  have 


xydp(x,y)  =  /  xd\x[x)  =  /  xdO(x)  =  0. 


Thus  IaxA  x  —  x')ydp(x,  y)  >  0  for  all  x'  G  Cx  and  all  A. 


Proposition  4.12.  Fix  a  game  satisfying  Assumption  f.l\  Let  k  >  0  be  even  and  X\, 
and  2/1 , ,  y2k  be  such  that: 

1.  Xi  G  Cx  and  y%  G  Cy  are  all  nonzero; 

2.  the  sequences  xi,  x3, . . . ,  x2k- i  and  yi,  y3, . . . ,  y2k-\  alternate  in  sign; 


0 

>  x2 k 


3.  x2i  =  x2i_\  and  y2i  =  y2l+  \  for  all  i  when  subscripts  are  interpreted  mod  2k. 

Then  p  =  Y2iti  an  extreme  correlated  equilibrium.  Furthermore,  all  finitely 

supported  extreme  correlated  equilibria  whose  support  does  not  contain  points  with  x  =  0  or 
y  =  0  are  of  this  form. 


Proof.  To  show  that  p  is  a  correlated  equilibrium  define  dn(x,y)  =  xydpjx ,  y).  Then  k  = 


Si=i  signal)  siga(yi)S^xi,yi)-  Defining  the  projection  kx  as  in  Proposition 


4.11 


we  have 


2k  k 

=  sign(^)  si§ n(Vi)Sxi  =  ^2  sign(x2i)  (sign(y2);)  +  sign(y2l-i))  SXM 

2= 1  2= 1 

k 

=  J^sign^Xo)^*  = 

2=1 


because  x2i  =  x2i_\  and  y2i  differs  in  sign  from  y2i_  1  by  assumption.  The  same  argument 
shows  that  ny  =  0,  so  /i  is  a  correlated  equilibrium. 

To  see  that  p  is  extreme,  suppose  p  =  p!  +  p"  where  p!  and  p"  are  correlated  equi¬ 
libria.  Clearly  p!  =  Y2i.ti  ai^(Xi,yi)  for  some  ai  >  0.  Define  da!  =  xydp'(x,y),  so  k '  = 
Y^itiaixiy^(Xi,yi)-  By  assumption 


2k 

k'x  ~  ^ 2X2i  {a2i-iy2i-l  +  Oi2iy2i)  SX2i 
2=1 

is  the  zero  measure.  Since  the  x2i  are  distinct  and  nonzero  we  must  have  a2i_\ y2i_ \  +  «2iy2  j  = 
0  for  all  i.  Similarly  since  n!y  =  0  we  have  a2i+1x2i+1  +  a2ix2i  =  0  for  all  i  (with  subscripts 
interpreted  mod  2k). 

The  Xi  and  y*  are  all  nonzero,  so  fixing  one  a*  fixes  all  the  others  by  these  equations. 
That  is  to  say,  these  equations  have  a  unique  solution  up  to  multiplication  by  a  scalar,  so 
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Figure  4:  The  support  of  an  extreme  correlated  equilibrium.  In  the  notation  of  Proposition 


4.12,  k  —  1,  X\  —  0.4,  x3  =  —0.6,  yi  =  0.2,  and  y3  =  —0.8. 


y!  is  a  positive  scalar  multiple  of  y.  But  the  splitting  y  —  y'  +  y"  was  arbitrary,  so  y  is 
extreme. 

A  similar  argument  shows  that  any  finitely  supported  correlated  equilibrium  y  whose 
support  does  not  contain  any  points  with  x  =  0  or  y  =  0  can  be  written  as  y  =  y’  +  y" 
where  y’  is  a  correlated  equilibrium  and  y"  is  a  correlated  equilibrium  of  the  above  form. 
Therefore  y  cannot  be  extreme  unless  it  is  of  this  form.  P 

Example  4.2  (cont’d).  For  some  examples  of  extreme  correlated  equilibria  of  games  of  this 
type,  see  Figures  [|  and  [5j 

To  count  the  number  of  extreme  correlated  equilibria  of  this  game  for  general  n  we  must 
count  the  number  of  essentially  different  sequences  of  ay  and  yi  of  the  type  mentioned  in 


Proposition  4.12  Fix  A:  and  let  k  =  2r  where  1  <  r  <  n.  Note  that  cyclically  shifting 
the  sequences  of  ay’s  and  yd s  by  two  does  not  change  y,  nor  does  reversing  the  sequence. 
Therefore  we  can  assume  without  loss  of  generality  that  X\ ,  y\  >  0.  We  then  have  n  possible 
choices  for  aq,a/i,£3,  and  y3,  n  —  1  possible  choices  for  a:5,  £7,  a/5,  and  y7,  etc.,  for  a  total  of 

possible  choices  of  the  Xi  and  yi.  These  will  always  be  essentially  different  (i.e.,  give 

rise  to  different  y)  unless  we  cyclically  permute  the  sequences  of  xt  and  y*  by  some  multiple 
of  four,  in  which  case  the  resulting  sequence  is  essentially  the  same.  The  number  of  such 
cyclic  permutations  is  r.  Therefore  the  total  number  of  extreme  correlated  equilibria  is 


e(n)  =  YI 


n\ 


— '  r  \  (n  —  r)\ 

=1  vv  ’ 

We  will  see  that  e(n)  =  0  (^(?r!)4).  That  is  to  say,  e(n)  is  asymptotically  upper  and 
lower  bounded  by  a  constant  times  -(n!)4.  The  expression  -(n!)4  is  just  the  final  term  in 
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Figure  5:  The  support  of  another  extreme  correlated  equilibrium.  In  the  notation  of  Propo¬ 


sition  [47T2J  k  —  2,  xi  —  0.4,  x3  =  —0.4,  x5  =  0.6,  x7  =  —0.6,  y7  =  0.6,  y3  =  —0.4,  y5  =  0.4, 
and  y7  =  —0.6. 


the  summation  for  e(n),  so  the  lower  bound  is  clear.  Define 


/  M  =  T 


e(n) 


n—1 

n 


-(n!)4  ^  n  —  s  (s!)4 

'  s=0  v  ' 


Then  f(n )  >  1  for  all  n.  We  will  now  show  that  f(n)  is  also  bounded  above.  Intuitively  this 
is  not  surprising  as  the  terms  in  the  summation  for  f{n)  die  off  extremely  fast  as  s  grows. 
For  all  1  <  s  <  n  -  1  we  have  that  the  ratio  of  term  s  +  1  in  the  summation  to  term  s  is: 


n— s—  1  ((s+1)!)4 


n  —  S 


1 

< 


—  ■Ai  n-s-  1  (s  +  l)4  —  8’ 

n—s  (s!)4  v  ' 

so  for  n  >  1  we  can  bound  the  sum  by  a  geometric  series: 

Ak  n  1  n  -A  1  8  n 

fin )  -  1  =  V - —  < 


^  16 

n  —  s  (s!)4  ~  n  —  1  ^  8*  7 (n  —  1)  ~  7 

s=l  v  >  t= o  v  ’ 


Therefore  1  <  /(n)  <  y  for  all  n,  so  e(n)  =  0  (^-(n!)4)  as  claimed.  Comparing  this 
to  the  results  of  the  previous  section  in  which  we  saw  that  the  number  of  extreme  Nash 
equilibria  of  this  game  is  n4 ,  we  see  that  in  this  case  there  is  a  super-exponential  separation 
between  the  number  of  extreme  Nash  and  the  number  of  extreme  correlated  equilibria.  This 
implies,  for  example,  that  computing  all  extreme  correlated  equilibria  is  not  an  efficient 
method  for  computing  all  extreme  Nash  equilibria,  even  though  all  extreme  Nash  equilibria 
are  extreme  correlated  equilibria  and  recognizing  whether  an  extreme  correlated  equilibrium 
is  an  extreme  Nash  equilibrium  is  easy.  There  are  simply  too  many  extreme  correlated 
equilibria. 
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Next  we  will  prove  a  more  abstract  version  of  Proposition  4.12  which  includes  certain 
extreme  points  which  are  not  finitely  supported.  First  we  need  a  brief  digression  to  ergodic 
theory.  The  first  definition  is  the  standard  definition  of  compatibility  between  a  measure  and 
a  transformation  on  a  space.  The  second  definition  expresses  one  notion  of  what  it  means  for 
a  transformation  to  “mix  up”  a  space  -  in  this  case  that  the  space  cannot  be  partitioned  into 
two  sets  of  positive  measure  which  do  not  interact  under  the  transformation.  Then  we  state 
the  main  ergodic  theorem  and  a  corollary  which  we  will  apply  to  exhibit  extreme  correlated 
equilibria  of  games  satisfying  Assumption  4.1| 


Definition  4.13.  Given  a  measure  g  G  A*  (S')  on  a  space  S,  a  measurable  function  g  :  S  — >  S 
is  called  (ju-)measure  preserving  if  /i(//_1(A))  =  g(A)  for  all  measurable  ACS.  Note  that 
if  g  is  invertible  (in  the  measure  theoretic  sense  that  an  almost  everywhere  inverse  exists), 
then  this  is  equivalent  to  the  condition  that  g(g(A))  =  g(A)  for  all  A. 


Definition  4.14.  Given  a  measure  g  G  A* (S'),  a  //-measure  preserving  transformation  g  is 
called  ergodic  if  g(A  A  g~1{A))  =  0  implies  g(A)  =  0  or  g(A)  =  g(S),  where  AAB  denotes 
the  symmetric  difference  {A  \  B)  U  (B  \  A). 


Example  4.15.  Fix  a  finite  set  S  and  a  permutation  g  :  S  — >  S.  Let  g  be  counting  measure 
on  S.  Then  g  is  measure  preserving.  Furthemore,  a  set  T  satisfies  g(g^1{T)  A  T)  =  0  if  and 
only  if  g_1(T)  =  T  if  and  only  if  T  is  a  union  of  cycles  of  g.  Therefore  g  is  ergodic  if  and 
only  if  it  consists  of  a  single  cycle. 

Example  4.16.  Fix  a  G  M.  Let  S  =  [0, 1)  and  let  g  be  Lebesgue  measure  on  S.  Define 
g  :  S  — >  S  by  g{x)  =  ( x  +  a)  mod  1  =  {x  +  a)  —  [x  +  a\ .  Then  g  is  //-measure  preserving 
because  Lebesgue  measure  is  translation  invariant.  It  can  be  shown  that  g  is  ergodic  if  and 
only  if  a  is  irrational.  For  a  proof  and  more  examples,  see  [26]. 

The  following  is  one  of  the  core  theorems  of  ergodic  theory.  We  will  only  use  it  to  prove 
the  corollary  which  follows,  so  it  need  not  be  read  in  detail.  The  proof  can  be  found  in  any 
text  on  ergodic  theory,  e.g.  [26]. 

Theorem  4.17  (Birkhoff’s  ergodic  theorem).  Fix  a  probability  measure  g  and  a  g-measure 
preserving  transformation  g.  Then  for  any  f  G  £}(g): 

•  f(x)  =  lirn^oc  L  Y^k=o  f(9n(x))  ex^sts  p-almost  everywhere, 

•  /  G  Cl{g), 

•  /  fdg  =  f  fdg, 

•  f(g(x))  =  f(x)  g-almost  everywhere,  and 

•  if  g  is  ergodic  then  f{x)  =  J  fdg  g-almost  everywhere. 

Corollary  4.18.  Suppose  //  and  v  are  probability  measures  such  that  v  is  absolutely  contin¬ 
uous  with  respect  to  // .  If  a  transformation  g  preserves  both  //  and  v  and  g  is  ergodic  with 
respect  to  g,  then  u  =  g. 
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Proof.  Fix  any  measurable  set  A.  Let  /  be  the  indicator  function  for  A,  i.e.  the  function 
equal  to  unity  on  A  and  zero  elsewhere.  Applying  Birkhoff’s  ergodic  theorem  to  /  and  p 
yields  f(x)  =  p(A)  //-almost  everywhere.  Since  v  is  absolutely  continuous  with  respect  to  p, 
f(x)  =  //(A)  //-almost  everywhere  also.  If  we  now  apply  Birkhoff’s  ergodic  theorem  to  v  we 
get: 


//(A)  =  /  fdv  =  /  fdv  =  /  p(A)dv  =  //(A) 


Hi 


Proposition  4.19.  Fix  measures  zq,  z/2, 1/3,  and  //4  G  A*((0, 1])  and  maps  f\  :  (0, 1]  — >  (0, 1] 
such  that  Ui+i  —  olo  fx1  (interpreting  subscripts  mod  A).  The  portion  of  the  measure  p  in 
the  ith  quadrant  of  I  x  I  will  be  constructed  in  terms  of  fi  and  zy.  Define  ji  :  (0, 1]  — >•  /  x  / 


J3O)  =  (-x,-f3(x)),  and  j4(x)  =  (f4(x),-x). 
4-1  is  satisfied,  supp|n|  C  Cx  x  and  G  £1(|k|) 


by  j i(x)  =  (xj^x)),  j2(x)  =  (~f2{x] 

Let  |/c|  =  Yhi=\ui°  h  ■  If  Assumption 
then  dp  =  |-A|d|n|  is  a  correlated  equilibrium. 

If  in  addition  f'4  o  /3  o  f2  o  f\  :  (0, 1]  — >  (0, 1]  is  ergodic  with  respect  to  v\,  then  p  is 
extreme. 


Proof.  First  we  must  show  that  p  is  a  correlated  equilibrium.  It  is  a  finite  measure  by  the 
assumption  G  £1(IKI)-  Define  g  :  I  x  / 


9(x,y)  =  < 


The  function  g  is  -measure  preserving.  To  see  this  £x  any  measurable  set  B  C  (0,1]  x 
(0,1].  Let  A  =  jf1(B).  Then  \n\(B)  =  |n|(A  x  (0,1])  =  zq(A)  by  definition  of  |/c|.  But 
=  g~\A  x  (0,1])  =  A  x  [-1,0),  so  \n\(g~\B))  =  |/c| (A  x  [-1,0))  =  ^(j-^A  x 
[—1,  0)))  =  z/4(/£1(A))  =  ui (A)  by  assumption.  Therefore  g  is  measure  preserving  for  subsets 
of  (0, 1]  x  (0, 1].  The  arguments  for  the  other  quadrants  are  similar  and  since  g  maps  each 
quadrant  into  a  different  quadrant,  g  is  measure  preserving  on  its  entire  domain. 


x  /  — >  /  x  /  as  follows. 

AO) 

if  x  >  0,  y  <  0 

h  (y) 

if  x  >  0,  y  >  0 

M~x) 

if  x  <  0,  y  >  0 

h{-y) 

if  x  <  0,  y  <  0 

arbitrary 

otherwise 

Using  the  terminology  of  Proposition  4.11,  dn  =  sign(x)  sign(//)d|ft|.  We  have  seen  that 
|/c|(A  x  (0, 1])  =  |k|(A  x  [—1,  0)),  so  k{A  x  (0, 1])  =  —  k{A  x  [—1,  0)).  Since  k(A  x  {0})  =  0, 
we  have  k(A  x  I)  =  0,  or  equivalently,  kx(A)  =  0.  A  similar  argument  implies  kx(A)  =  0  if 
A  C  [—1,0).  Clearly  ^({0})  =  0  by  dehnition  of  kx,  so  kx  is  the  zero  measure.  A  similar 
argument  shows  that  Ky  is  the  zero  measure,  so  p  is  a  correlated  equilibrium  by  Proposition 

mm 

Now  we  will  show  via  several  steps  that  p  is  extreme.  Write  p  —  pi  +  p2  where  the  pt 
are  nonzero  correlated  equilibria.  Since  these  are  all  positive  measures,  the  pt  are  absolutely 
continuous  with  respect  to  p.  Define  d\Ki\  =  \xy\dpi. 

Next  we  show  that  g  is  (-measure  preserving.  We  will  demonstrate  this  fact  for  B  C 
(0,1]  x  (0,1].  As  above,  we  define  A  =  jf1(B).  Then  \ni\(B)  =  |/c*|(A  x  (0,1])  since 
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(. A  x  (0,1])  A  B  has  |/c|  measure  zero  and  |/Cj|  is  absolutely  continuous  with  respect  to 
|/c|.  Furthermore,  \Ki\(g~l(B))  =  |/Cj|(A  x  [—1,0)).  But  g%  is  a  correlated  equilibrium  so 
Ki(A  x  (0, 1])  =  -Ki(A  x  [-1,0)).  Hence  |/Ci|(^_1(5))  =  \mh\(A  x  [-1,0))  =  \Ki\(A  x  (0, 1])  = 
\ni\(yB).  Again,  the  proof  is  the  same  for  B  contained  in  other  quadrants,  so  g  is  fq [-measure 
preserving. 

For  the  second- to-last  step  we  prove  that  g  is  ergodic  with  respect  to  \k\.  Suppose 
B  C  /  x  /  is  such  that  \n\(g~1(B)  A  B)  —  0.  Let  Qi  be  the  intersection  of  B  with  the 
ith  quadrant.  Then  |«|(<7-1(<2i+i)  A  Qi)  =  0,  so  \n\(g~A(Q\)  A  Qi)  =  0.  Let  A  =  jf1(Q i). 
Then  \^\{g~A{Qi)  A  Qi)  =  zq((/4  °  /3  °  fi  °  /i)_1(^)  A  A)  =  0.  By  assumption  the  map 
/4o/3o/2°  /i  is  ergodic,  so  zq(A)  =  0  or  ^i(A)  =  iq((0, 1])  =  |/c|((0, 1]  x  (0, 1]).  Therefore 
IkI(Qi)  =  ui(A)  =  0  or  | ac| (<3i.)  =  |«|((0,1]  x  (0,1]).  In  either  case  since  g  is  k] -measure 
preserving  we  get  |/c|(Qi)  =  |/c|(<5i)  for  all  i.  Therefore  \n\(B)  =  0  or  \n\(B)  =  |/c| (/  x  /),  so 
g  is  ergodic  with  respect  to  \k\. 

Normalizing  |ac|  and  \h:t  \  to  be  probability  measures,  we  can  apply  Corollary |4.18| to  obtain 
|«j|  =  ^p|(/xq  I  A-  By  assumption  the  set  on  which  \xy\  is  zero  has  /i  measure  zero.  Therefore 


dg-i  = 


«i|(/x/)  1 
l«l (I  x  !)  \xy | 


d\n 


«i|(/x/) 
]«](/  x  I) 


dg,, 


so  fa 


|k|(/x/) 


/i  and  /i  is  extreme. 


□ 


Above  we  have  constructed  g  and  g  so  that  g  maps  the  quadrants  counter-clockwise  - 
quadrant  1  to  quadrant  2,  etc.  However,  the  same  argument  would  go  through  if  g  mapped 
the  quadrants  clockwise. 


To  view  Proposition  4.12  as  a  special  case  of  Proposition  4.19,  let  each  u,  be  a  uniform 
probability  measure  over  a  finite  subset  of  (0,1].  The  function  g  is  defined  by  g(xi,yi )  = 
(xj+i, Vi- |_i)  and  the  are  defined  to  be  compatible  with  this.  The  map  /40/30/,0  f\  is  a 
permutation  on  the  support  of  U\ ,  which  is  precisely  the  positive  values  of  Xi.  By  construction 
this  permutation  consists  of  a  single  cycle,  hence  it  is  ergodic. 

Exarnple  4.3  (cont’d).  We  can  combine  Example |4.16|and  Proposition|4.19|to  exhibit  extreme 
points  of  the  set  of  correlated  equilibria  for  this  game  which  are  not  finitely  supported.  Let 
0  <  a  <  b  <  1.  Let  zy  be  Lebesgue  measure  on  [a,  b)  for  all  i.  Fix  a  such  that  ^  is  irrational. 
Define  j\  :  [a,  b )  —>  [a,  b)  by  f{x)  =  (x  —  a  +  a  mod  {b  —  a))  +  a.  This  is  just  an  affinely 
scaled  version  of  Example  4.16  so  f\  is  ^-measure  preserving  and  ergodic.  Define  f±  on 


(0, 1]  \  [a,  b)  arbitrarily,  because  that  is  a  set  of  measure  zero.  Let  fo,  fz,  .A  :  (0, 1]  — >  (0, 1]  be 


the  identity.  These  data  satisfy  all  the  assumptions  of  Proposition  |4.19[  In  particular,  since 
0  <  a  <  b  <  1,  xy  is  bounded  away  from  zero  on  the  support  of  |ac|.  Therefore  G  £1(|k|). 
Since  zy  is  not  finitely  supported,  g  is  an  extreme  correlated  equilibrium  which  is  not  finitely 
supported.  The  support  of  g  is  shown  in  Figure  [6]  with  parameters  a  =  0.2,  b  =  0.8,  and 

a  =  7S- 


Definition  4.4.  Given  a  compact  Hausdorff  space  K  we  say  that  a  set  of  measures  M.  C  A (K) 
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1 


Figure  6:  The  support  set  of  an  extreme  correlated  equilibrium  which  is  not  finitely  sup¬ 
ported.  Extremality  of  this  equilibrium  depends  sensitively  on  the  choices  of  endpoints  for  the 
line  segments.  In  the  case  pictured  there  are  segments  connecting:  (0.2,  —0.2)  to  (0.8,  —0.8); 

to 


(0.8- ^,0.8); 


(-0.2, -0.2)  to  (-0.8, -0.8);  (-0.2, 0.2)  to  (-0.8,  0.8);  (o.2,0.2+Tg 


and  (0.8  -  ^ ,  0.2J  to  (0.8,  0.2  +  ^=J  . 


is  describable  by  moments  if  there  exists  an  integer  d,  continuous  maps  gi, . . . ,  :  K  — *• 
M,  and  a  set  M  C  W1  such  that  a  measure  p  is  in  Ai  if  and  only  if  ( J  gidp, . . . ,  J  gddp )  G  M. 

The  results  of  [21]  show  that  the  maximin  and  minimax  strategy  sets  of  a  two-player  zero- 
sum  polynomial  game  can  always  be  described  by  moments.  Introducing  a  similar  notion  for 
n-tuples  of  moments,  the  set  of  Nash  equilibria  can  always  be  described  by  moments  in  any 
polynomial  game  m  However,  combining  this  example  with  the  following  proposition  we 
see  that  the  set  of  correlated  equilibria  of  a  polynomial  game  cannot  in  general  be  described 
by  moments. 

Proposition  4.5.  Let  At  C  A *(K)  be  a  set  of  measures  describable  by  moments.  Then 
all  extreme  points  of  At  have  finite  support  and  this  support  is  uniformly  bounded  by  d  +  1, 
where  d  is  the  integer  associated  with  the  description  of  Ad  by  moments. 

Proof.  Suppose  there  exists  a  measure  p  G  A4  which  is  extreme  and  supported  on  more  than 
d  +  1  points,  so  we  can  partition  the  domain  of  p  into  d  +  2  sets  Bi, ... ,  Bd+ 2  of  positive 
measure.  Then  p  is  a  convex  combination  of  the  measures  (d  +  2)/i|  but  only  if  all  the 
weights  in  the  convex  combination  are  equal  to  >  0.  The  moments  of  p  are  a  convex 
combination  of  the  moments  of  these  measures.  Since  there  are  d  moments,  there  exist 
convex  combinations  of  at  most  d  +  1  of  the  measures  (d  +  2 )p\b1  with  the  same  moments 
as  p  by  Caratheodory’s  theorem.  We  can  write  p  as  a  strict  convex  combination  of  such 
measures,  contradicting  extremality  of  p.  □ 
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5  Conclusions 


We  have  proven  several  new  characterizations  of  correlated  equilibria  in  continuous  games, 
and  applied  them  to  compute  correlated  equilibria,  both  algorithmically  and  analytically. 
This  has  allowed  us  to  prove  that  in  general  the  set  of  correlated  equilibria  of  a  polynomial 
game  cannot  be  described  exactly  in  terms  of  finitely  many  moments.  Nonetheless,  a  sample 
correlated  equilibrium  or  the  entire  set  of  correlated  equilibria  can  be  approximated  within 
an  arbitrary  degree  of  accuracy  by  a  sequence  of  semidehnite  programs. 

These  results  leave  several  open  questions.  If  we  define  a  moment  map  to  be  any  map 
of  the  form  it  t— > ►  (/  fidir,  f  /2chr, . . . ,  /fed7r),  then  we  have  shown  that  the  set  of  correlated 
equilibria  is  not  the  inverse  image  of  any  set  under  any  moment  map.  On  the  other  hand, 
since  moment  maps  are  linear  and  weak*  continuous,  we  know  that  the  image  of  the  set  of 
correlated  equilibria  under  any  moment  map  is  convex  and  compact.  Supposing  the  utilities 
and  the  /,  are  polynomials,  is  there  anything  more  we  can  say  about  this  image?  In  particular, 
is  it  semialgebraic  (i.e.,  describable  in  terms  of  finitely  many  polynomial  inequalities)? 

For  any  continuous  game,  the  set  of  correlated  equilibria  is  nonempty,  and  this  can  be 
proven  constructively  as  in  H2j.  Under  the  same  assumptions  we  can  prove  the  existence 
of  a  Nash  equilibrium,  but  the  proof  is  nonconstructive,  or  at  least  does  not  seem  to  give 
an  efficient  algorithm  for  constructing  an  equilibrium  123-  In  the  case  of  polynomial  games, 
existence  of  a  Nash  equilibrium  immediately  gives  existence  of  a  finitely  supported  Nash 
equilibrium  by  Caratheodory’s  theorem,  which  is  constructive  [57].  Therefore  there  exists 
a  finitely  supported  correlated  equilibrium  of  any  polynomial  game.  Is  there  a  construc¬ 
tive  way  to  prove  this  fact  directly,  without  going  through  Nash  equilibria?  Such  a  proof 
could  potentially  lead  to  a  provably  efficient  algorithm  for  computing  a  sample  correlated 
equilibrium  of  a  polynomial  game. 

Finally,  we  note  that  while  the  adaptive  discretization  and  moment  relaxation  algorithms 
converge  in  general  and  work  well  in  practice,  we  do  not  know  of  any  results  regarding  rate 
of  convergence.  If  we  regard  the  probability  distributions  produced  by  these  algorithms  at 
the  kth  iteration  as  efc-correlated  equilibria,  how  fast  does  ek  converge  to  zero? 
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A  Semidefinite  programming,  sums  of  squares,  and 
moments  of  measures 

Definition  A.l.  A  semidefinite  program  is  an  optimization  problem  of  the  form: 

minimize  L(S) 
subject  to  T(S)  =  v 

S'  is  a  symmetric  matrix 
S  >z  0  (positive  semidefinite), 

where  L  is  a  given  linear  functional,  T  is  a  given  linear  transformation,  v  is  a  given  vector, 
and  S  is  a  square  matrix  of  decision  variables. 

Semidefinite  programs  are  convex  and  generalize  linear  programs  (T  and  v  can  be  designed 
to  make  S  diagonal,  in  which  case  the  condition  S'  ^  0  is  the  same  as  the  condition  that  S  >  0 
elementwise).  Many  problems  can  be  expressed  exactly  or  approximately  as  semidefinite 
programs,  and  this  is  important  because  semidefinite  programs  can  be  solved  efficiently  by 
interior  point  methods.  For  details  and  a  variety  of  examples  see  [3D]  and  [2D] . 

The  square  of  a  real-valued  function  is  nonnegative  on  its  entire  domain,  as  is  a  sum  of 
squares  of  real- valued  functions.  In  particular,  any  polynomial  of  the  form  p(x)  =  Jfpl(x), 
where  pk  are  polynomials,  is  guaranteed  to  be  nonnegative  for  all  x.  This  gives  a  sufficient 
condition  for  a  polynomial  to  be  nonnegative.  It  is  a  classical  result  that  this  condition  is 
also  necessary  if  p  is  univariate  [23] . 

Proposition  A. 2.  A  univariate  polynomial  p  is  nonnegative  on  M  if  and  only  if  it  is  a  sum 
of  squares. 

Proof.  A  simpler  version  of  the  proof  of  the  following  proposition.  □ 

Proposition  A. 3  (Markov-Lukacs  [15]).  A  univariate  polynomial  p(x)  is  nonnegative  on 
the  interval  [—1, 1]  if  and  only  if  p(x)  =  s(x)  +  (1  —  x2)t(x)  where  s  and  t  are  both  sums  of 
squares  of  polynomials. 

Proof.  Direct  algebraic  manipulations  show  that  the  set  of  polynomials  of  the  form  s(x)  +  (l  — 
x2)t(x)  where  s  and  t  are  sums  of  squares  of  polynomials  in  x  is  closed  under  multiplication 
and  contains  all  polynomials  of  the  following  forms:  a  for  a  >  0,  (x  —  a)2  +  b 2  for  a,  b  G  R, 
x  —  a  for  a  <  —  1,  and  a  —  x  for  a  >  1.  By  assumption  p{x)  factors  as  a  product  of  terms  of 
these  types,  because  any  real  root  of  p  in  the  interval  (—1, 1)  must  have  even  multiplicity.  D 

These  sum  of  squares  conditions  are  easy  to  express  using  linear  equations  and  semidefi¬ 
nite  constraints. 

Proposition  A. 4.  A  univariate  polynomial  p(x)  =  Y(k=o  Pkxk  of  degree  at  most  2 d  is  a  sum 
of  squares  of  polynomials  if  and  only  if  there  exists  a  symmetric  positive  semidefinite  matrix 
Q  G  M(d+1)x(d+1)  such  that  pk  =  J2i+j=kQij  (‘^umbering  the  rows  and  columns  of  Q  from  0 
to  d). 
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Proof.  Relating  the  coefficients  of  p(x)  to  the  entries  of  Q  in  this  way  is  the  same  as  writing 
p{x)  =  x7  Qx  where  x  =  [l  x  x2  ■  ■  ■  .  Thought  of  in  this  way,  saying  that  p(x)  is 

a  sum  of  squares  is  the  same  as  saying  that  Q  =  JA  q,(fi  for  some  column  vectors  g*  and 
in  this  case  Q  is  clearly  positive  semidefinite.  Conversely,  if  Q  is  positive  semidefinite  then 
there  exists  a  matrix  F  such  that  Q  =  FTF ,  so  p(x)  =  xTQx  =  JA  [Fx]-.  □ 

Similar  semidehnite  characterizations  exist  for  multivariate  polynomials  to  be  sums  of 
squares.  While  the  condition  of  being  a  sum  of  squares  does  not  characterize  general  non¬ 
negative  multivariate  polynomials  exactly,  there  exist  sequences  of  sum  of  squares  relaxations 
which  can  approximate  the  set  of  nonnegative  polynomials  (on  e.g.  Mfc,  [—1,  l]fc,  or  a  more 
general  semialgebraic  set)  arbitrarily  tightly  [23j.  Furthermore,  for  some  special  classes  of 
multivariate  polynomials,  the  sum  of  squares  condition  is  exact. 

Proposition  A. 5.  A  matrix  M(t )  whose  entries  are  univariate  polynomials  in  t  is  positive 
semidefinite  on  [—1, 1]  if  and  only  if  x'M(t)x  =  S(x,t)  +  (1  —  t2)T(x,t )  where  S  and  T  are 
polynomials  which  are  sums  of  squares. 

Now  suppose  we  wish  to  answer  the  question  of  whether  a  finite  sequence  (/i°, . . . ,  pk)  of 
reals  correspond  to  the  moments  of  a  measure  on  [—1,1],  i.e.  whether  there  exists  a  positive 
measure  p  on  [—1,1]  such  that  pl  =  f  xldp(x).  Clearly  if  such  a  measure  exists  then  we 
must  have  f  p(x)dp(x)  >  0  for  any  polynomial  p  of  degree  at  most  k  which  is  nonnegative 
on  [—1, 1],  This  necessary  condition  for  moments  to  correspond  to  a  measure  turns  out  to 
be  sufficient  m  and  can  be  written  in  terms  of  semidefinite  constraints. 

Proposition  A. 6.  The  condition  that  a  finite  sequence  of  numbers  ( p° , . . . ,  pk)  be  the  mo¬ 
ments  of  a  positive  measure  on  [—1, 1]  can  be  written  in  terms  of  linear  equations  and  semidef¬ 
inite  matrix  constraints. 

One  can  formulate  similar  questions  about  whether  a  finite  sequence  of  numbers  cor¬ 
responds  to  the  joint  moments  f  xf  ■■■xlfdp(x)  of  a  positive  measure  p  on  [ — 1,  l]fc  (or  a 
more  general  semialgebraic  set).  Using  a  sequence  of  semidehnite  relaxations  of  the  set  of 
nonnegative  polynomials  on  [—1,  l]fc,  a  sequence  of  necessary  conditions  for  joint  moments 
is  obtained.  These  conditions  approximate  the  set  of  joint  moments  arbitrarily  closely. 
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